Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 1.829100806s FAIL (2025-08-11 04:57:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 04:44:24.932296741 (59h18m19.363376862s) [upload=61797]: Starting upload Aug 11 04:44:24.932375572 (59h18m19.363455703s) [upload=61797]: Done starting upload Aug 11 04:44:24.932485803 (59h18m19.363565994s) [upload=61797]: Uploading segments... Aug 11 04:44:24.932550086 (59h18m19.363630247s) [upload=61797]: Got next segment. Inline: true Aug 11 04:44:24.932557180 (59h18m19.363637331s) [upload=61797]: Waiting for error group managing segments... Aug 11 04:44:24.932610262 (59h18m19.363690453s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:44:24.932616814 (59h18m19.363696995s) [upload=61797]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:44:24.932621844 (59h18m19.363702005s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:44:25.294615282 (59h18m19.725695413s) [upload=61797]: Done uploading segments... Aug 11 04:44:25.294637325 (59h18m19.725717456s) [upload=61797]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.829131434s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 7.926890925s FAIL (2025-08-11 04:57:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.926926363s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........................XX..................X.....X..X....] duration: 7.345972398s FAIL (2025-08-11 04:57:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 04:53:15.975483199 (59h27m10.40656332s) [upload=61957]: Starting upload Aug 11 04:53:15.975529377 (59h27m10.406609498s) [upload=61957]: Done starting upload Aug 11 04:53:15.975620010 (59h27m10.406700141s) [upload=61957]: Uploading segments... Aug 11 04:53:15.975802861 (59h27m10.406883002s) [upload=61957]: Got next segment. Inline: false Aug 11 04:53:15.975845712 (59h27m10.406925843s) [upload=61957]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:53:15.975853518 (59h27m10.406933689s) [upload=61957]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:53:16.197092986 (59h27m10.628173157s) [upload=61957]: Next segment err: metaclient: Too Many Requests Aug 11 04:53:16.197121811 (59h27m10.628201952s) [upload=61957]: Done uploading segments... Aug 11 04:53:16.197141379 (59h27m10.62822157s) [upload=61957]: Upload finished. err: metaclient: Too Many Requests Duration: 7.346002655s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 1.920143648s FAIL (2025-08-11 04:56:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 04:44:24.932296741 (59h18m19.363376862s) [upload=61797]: Starting upload Aug 11 04:44:24.932375572 (59h18m19.363455703s) [upload=61797]: Done starting upload Aug 11 04:44:24.932485803 (59h18m19.363565994s) [upload=61797]: Uploading segments... Aug 11 04:44:24.932550086 (59h18m19.363630247s) [upload=61797]: Got next segment. Inline: true Aug 11 04:44:24.932557180 (59h18m19.363637331s) [upload=61797]: Waiting for error group managing segments... Aug 11 04:44:24.932610262 (59h18m19.363690453s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:44:24.932616814 (59h18m19.363696995s) [upload=61797]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:44:24.932621844 (59h18m19.363702005s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:44:25.294615282 (59h18m19.725695413s) [upload=61797]: Done uploading segments... Aug 11 04:44:25.294637325 (59h18m19.725717456s) [upload=61797]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.920172383s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........................XX..................X.....X..X...] duration: 7.731014224s FAIL (2025-08-11 04:56:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 04:53:15.975483199 (59h27m10.40656332s) [upload=61957]: Starting upload Aug 11 04:53:15.975529377 (59h27m10.406609498s) [upload=61957]: Done starting upload Aug 11 04:53:15.975620010 (59h27m10.406700141s) [upload=61957]: Uploading segments... Aug 11 04:53:15.975802861 (59h27m10.406883002s) [upload=61957]: Got next segment. Inline: false Aug 11 04:53:15.975845712 (59h27m10.406925843s) [upload=61957]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:53:15.975853518 (59h27m10.406933689s) [upload=61957]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:53:16.197092986 (59h27m10.628173157s) [upload=61957]: Next segment err: metaclient: Too Many Requests Aug 11 04:53:16.197121811 (59h27m10.628201952s) [upload=61957]: Done uploading segments... Aug 11 04:53:16.197141379 (59h27m10.62822157s) [upload=61957]: Upload finished. err: metaclient: Too Many Requests Duration: 7.731040633s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 7.501016019s FAIL (2025-08-11 04:56:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.501046767s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 8.433246795s FAIL (2025-08-11 04:55:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.433271603s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [X............................................X..............] duration: 1.828701319s FAIL (2025-08-11 04:55:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 04:44:24.932296741 (59h18m19.363376862s) [upload=61797]: Starting upload Aug 11 04:44:24.932375572 (59h18m19.363455703s) [upload=61797]: Done starting upload Aug 11 04:44:24.932485803 (59h18m19.363565994s) [upload=61797]: Uploading segments... Aug 11 04:44:24.932550086 (59h18m19.363630247s) [upload=61797]: Got next segment. Inline: true Aug 11 04:44:24.932557180 (59h18m19.363637331s) [upload=61797]: Waiting for error group managing segments... Aug 11 04:44:24.932610262 (59h18m19.363690453s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:44:24.932616814 (59h18m19.363696995s) [upload=61797]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:44:24.932621844 (59h18m19.363702005s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:44:25.294615282 (59h18m19.725695413s) [upload=61797]: Done uploading segments... Aug 11 04:44:25.294637325 (59h18m19.725717456s) [upload=61797]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.828721228s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.X............................................X.............] duration: 1.807021018s FAIL (2025-08-11 04:55:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 04:44:24.932296741 (59h18m19.363376862s) [upload=61797]: Starting upload Aug 11 04:44:24.932375572 (59h18m19.363455703s) [upload=61797]: Done starting upload Aug 11 04:44:24.932485803 (59h18m19.363565994s) [upload=61797]: Uploading segments... Aug 11 04:44:24.932550086 (59h18m19.363630247s) [upload=61797]: Got next segment. Inline: true Aug 11 04:44:24.932557180 (59h18m19.363637331s) [upload=61797]: Waiting for error group managing segments... Aug 11 04:44:24.932610262 (59h18m19.363690453s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:44:24.932616814 (59h18m19.363696995s) [upload=61797]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:44:24.932621844 (59h18m19.363702005s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:44:25.294615282 (59h18m19.725695413s) [upload=61797]: Done uploading segments... Aug 11 04:44:25.294637325 (59h18m19.725717456s) [upload=61797]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.807046938s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................XX..................X.....X..X..] duration: 8.202130899s FAIL (2025-08-11 04:55:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 04:53:15.975483199 (59h27m10.40656332s) [upload=61957]: Starting upload Aug 11 04:53:15.975529377 (59h27m10.406609498s) [upload=61957]: Done starting upload Aug 11 04:53:15.975620010 (59h27m10.406700141s) [upload=61957]: Uploading segments... Aug 11 04:53:15.975802861 (59h27m10.406883002s) [upload=61957]: Got next segment. Inline: false Aug 11 04:53:15.975845712 (59h27m10.406925843s) [upload=61957]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:53:15.975853518 (59h27m10.406933689s) [upload=61957]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:53:16.197092986 (59h27m10.628173157s) [upload=61957]: Next segment err: metaclient: Too Many Requests Aug 11 04:53:16.197121811 (59h27m10.628201952s) [upload=61957]: Done uploading segments... Aug 11 04:53:16.197141379 (59h27m10.62822157s) [upload=61957]: Upload finished. err: metaclient: Too Many Requests Duration: 8.202156669s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 7.821284056s FAIL (2025-08-11 04:54:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.821304446s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..X............................................X............] duration: 1.818161576s FAIL (2025-08-11 04:54:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 04:44:24.932296741 (59h18m19.363376862s) [upload=61797]: Starting upload Aug 11 04:44:24.932375572 (59h18m19.363455703s) [upload=61797]: Done starting upload Aug 11 04:44:24.932485803 (59h18m19.363565994s) [upload=61797]: Uploading segments... Aug 11 04:44:24.932550086 (59h18m19.363630247s) [upload=61797]: Got next segment. Inline: true Aug 11 04:44:24.932557180 (59h18m19.363637331s) [upload=61797]: Waiting for error group managing segments... Aug 11 04:44:24.932610262 (59h18m19.363690453s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:44:24.932616814 (59h18m19.363696995s) [upload=61797]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:44:24.932621844 (59h18m19.363702005s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:44:25.294615282 (59h18m19.725695413s) [upload=61797]: Done uploading segments... Aug 11 04:44:25.294637325 (59h18m19.725717456s) [upload=61797]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.818180543s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................XX..................X.....X..X.] duration: 7.0507575s FAIL (2025-08-11 04:54:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 04:53:15.975483199 (59h27m10.40656332s) [upload=61957]: Starting upload Aug 11 04:53:15.975529377 (59h27m10.406609498s) [upload=61957]: Done starting upload Aug 11 04:53:15.975620010 (59h27m10.406700141s) [upload=61957]: Uploading segments... Aug 11 04:53:15.975802861 (59h27m10.406883002s) [upload=61957]: Got next segment. Inline: false Aug 11 04:53:15.975845712 (59h27m10.406925843s) [upload=61957]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:53:15.975853518 (59h27m10.406933689s) [upload=61957]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:53:16.197092986 (59h27m10.628173157s) [upload=61957]: Next segment err: metaclient: Too Many Requests Aug 11 04:53:16.197121811 (59h27m10.628201952s) [upload=61957]: Done uploading segments... Aug 11 04:53:16.197141379 (59h27m10.62822157s) [upload=61957]: Upload finished. err: metaclient: Too Many Requests Duration: 7.050786046s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 8.870548965s FAIL (2025-08-11 04:53:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.870575335s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...X............................................X...........] duration: 2.11583443s FAIL (2025-08-11 04:53:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 04:44:24.932296741 (59h18m19.363376862s) [upload=61797]: Starting upload Aug 11 04:44:24.932375572 (59h18m19.363455703s) [upload=61797]: Done starting upload Aug 11 04:44:24.932485803 (59h18m19.363565994s) [upload=61797]: Uploading segments... Aug 11 04:44:24.932550086 (59h18m19.363630247s) [upload=61797]: Got next segment. Inline: true Aug 11 04:44:24.932557180 (59h18m19.363637331s) [upload=61797]: Waiting for error group managing segments... Aug 11 04:44:24.932610262 (59h18m19.363690453s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:44:24.932616814 (59h18m19.363696995s) [upload=61797]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:44:24.932621844 (59h18m19.363702005s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:44:25.294615282 (59h18m19.725695413s) [upload=61797]: Done uploading segments... Aug 11 04:44:25.294637325 (59h18m19.725717456s) [upload=61797]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.115876962s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................XX..................X.....X..X] duration: 930.901021ms FAIL (2025-08-11 04:53:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 04:53:15.975483199 (59h27m10.40656332s) [upload=61957]: Starting upload Aug 11 04:53:15.975529377 (59h27m10.406609498s) [upload=61957]: Done starting upload Aug 11 04:53:15.975620010 (59h27m10.406700141s) [upload=61957]: Uploading segments... Aug 11 04:53:15.975802861 (59h27m10.406883002s) [upload=61957]: Got next segment. Inline: false Aug 11 04:53:15.975845712 (59h27m10.406925843s) [upload=61957]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:53:15.975853518 (59h27m10.406933689s) [upload=61957]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:53:16.197092986 (59h27m10.628173157s) [upload=61957]: Next segment err: metaclient: Too Many Requests Aug 11 04:53:16.197121811 (59h27m10.628201952s) [upload=61957]: Done uploading segments... Aug 11 04:53:16.197141379 (59h27m10.62822157s) [upload=61957]: Upload finished. err: metaclient: Too Many Requests Duration: 930.920418ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 8.364843381s FAIL (2025-08-11 04:53:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.364870062s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 18.222129035s FAIL (2025-08-11 04:52:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 18.222242121s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [X..............................XX..................X.....X..] duration: 14.613047663s FAIL (2025-08-11 04:52:20 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 11 04:50:04.654122771 (59h23m59.085202902s) [upload=61903]: Starting upload Aug 11 04:50:04.654167006 (59h23m59.085247137s) [upload=61903]: Done starting upload Aug 11 04:50:04.654293087 (59h23m59.085373218s) [upload=61903]: Uploading segments... Aug 11 04:50:04.654458323 (59h23m59.085538454s) [upload=61903]: Got next segment. Inline: false Aug 11 04:50:04.654484242 (59h23m59.085564373s) [upload=61903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:50:04.654489162 (59h23m59.085569293s) [upload=61903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:50:05.098864103 (59h23m59.529944254s) [seg_pos={0 0}, upload=61903]: Begin upload segment... Aug 11 04:50:05.098932725 (59h23m59.530012866s) [seg_pos={0 0}, upload=61903]: Done begin upload segment. Aug 11 04:50:05.098964245 (59h23m59.530044386s) [noise=true , address=104.36.82.47:4537 , piece_id=4KQCMB7R56S6EKMXLF2E7FWAWPOG3LC2KYHFIKOSVWYRVRESYWAQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.099043326 (59h23m59.530123487s) [noise=true , address=104.36.82.59:4328 , piece_id=UPVR344E5YWFRFQI5LWHPTQFFJU2VBT6V7EFEAB5TNMML5L5WYNQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.099548633 (59h23m59.530628814s) [noise=true , address=104.36.82.39:4755 , piece_id=AWVSJKMX5FP4NURUUGJ3S7MLQM2L67AYEFAERTHZ7RCXZC7ORMAQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.099872773 (59h23m59.530952904s) [noise=true , address=104.36.82.46:4487 , piece_id=7AKA2FLAKZKFGRAWVMLUBCDGLEVSKVAWJZY4EVKZKII4JQJWI7IQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.099991821 (59h23m59.531071952s) [noise=true , address=104.36.82.59:4321 , piece_id=H7R47I3JTPUGN6FZC3RWZQYUACTALXGAY5YLSEHNHRFMF5WPZNCA, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.100415351 (59h23m59.531495492s) [noise=true , address=104.36.82.23:4453 , piece_id=DDI55E5BTVW3MPNJSOMSA4IGQCOA5WQ2PZHS2VVM2SOLWJMVFGLQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.100426312 (59h23m59.531506443s) [noise=true , address=104.36.82.60:4360 , piece_id=EOHB45DR43LHMLLCOUHSWPJ2NUDWJJO3UOHTCYLEVXLNKYAOK4CQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.100818353 (59h23m59.531898494s) [noise=true , address=104.36.82.12:4167 , piece_id=MGBXFYX5DYMVKEWLKWUASKICDFTG23M26ZCODE4F7IQFG2UVQH6Q, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.101245610 (59h23m59.532325751s) [noise=true , address=104.36.82.8:4978 , piece_id=JFT7KYLTNEP3WVBHGY52MWSLXVFP47EWVTQCFEEXO3VTMIMDLJPQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.101600088 (59h23m59.532680219s) [noise=true , address=104.36.82.37:4666 , piece_id=QGHN6WDXGXHE5ONVRDK3RU2PGGIBBYHYKPXLUIW4DH3CM7DHE24Q, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.102050571 (59h23m59.533130712s) [noise=true , address=104.36.82.55:4199 , piece_id=C5HHQYRNV4GJJDETVYB6EQUVQYPLEDOHTNHFJGOJABSJTXQUQRAA, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.102458752 (59h23m59.533538883s) [noise=true , address=104.36.82.58:4282 , piece_id=5KOWL5AF4VBFWRHLFCR3HPZT4O3FZ2N2XBFOSU6UVFT54GL3S34A, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.102775637 (59h23m59.533855768s) [noise=true , address=104.36.82.45:4907 , piece_id=I4QDBJDKUIWFAD7BKMWFJSD75I24R2DLKORKCUQ6P7BQVLVLGL2A, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.103089639 (59h23m59.53416977s) [noise=true , address=64.49.13.27:4961 , piece_id=4Q6BHP3IMIMSBP6XDATZCNUUVINRH4N7AKRCTCSOAQVJY4ZGCLIQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.103547073 (59h23m59.534627214s) [noise=true , address=104.36.82.16:4305 , piece_id=OZJMNIZFC3P6ZMFPJ2TCC5ARRMSEZ2Z4FAWXGHNVPXJ67RYFC7TQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.103718020 (59h23m59.534798161s) [noise=true , address=104.36.82.62:4419 , piece_id=JA7GHSJA24CQS2WDSD6V4Z6HZ6G5Y64R7UHJEZJBQLNBTEFFQVVQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.104095302 (59h23m59.535175433s) [noise=true , address=104.36.82.14:4247 , piece_id=N4AEYFHDVO2BO2ZHBZ3MLCTP7OT7WK5QQU7FRAANSMFGWZHXM3DA, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.104403903 (59h23m59.535484044s) [noise=true , address=104.36.82.13:4197 , piece_id=NWUXYJSX2CWINMKSEWPJPESDCDJDNK6RG7DOSXSVROZHAZ4LFR3Q, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:06.295215072 (59h24m0.726295203s) [upload=61903]: Next returned nil segment Aug 11 04:50:06.295232645 (59h24m0.726312776s) [upload=61903]: Waiting for error group managing segments... Aug 11 04:50:06.691486175 (59h24m1.122566296s) [noise=true , address=64.49.13.27:4961 , piece_id=4Q6BHP3IMIMSBP6XDATZCNUUVINRH4N7AKRCTCSOAQVJY4ZGCLIQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:07.192517489 (59h24m1.62359762s) [noise=true , address=104.36.82.45:4907 , piece_id=I4QDBJDKUIWFAD7BKMWFJSD75I24R2DLKORKCUQ6P7BQVLVLGL2A, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:07.301483609 (59h24m1.73256373s) [noise=true , address=104.36.82.58:4282 , piece_id=5KOWL5AF4VBFWRHLFCR3HPZT4O3FZ2N2XBFOSU6UVFT54GL3S34A, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:07.515352383 (59h24m1.946432504s) [noise=true , address=104.36.82.46:4487 , piece_id=7AKA2FLAKZKFGRAWVMLUBCDGLEVSKVAWJZY4EVKZKII4JQJWI7IQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:07.774866883 (59h24m2.205947014s) [noise=true , address=104.36.82.39:4755 , piece_id=AWVSJKMX5FP4NURUUGJ3S7MLQM2L67AYEFAERTHZ7RCXZC7ORMAQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:08.284029245 (59h24m2.715109366s) [noise=true , address=104.36.82.37:4666 , piece_id=QGHN6WDXGXHE5ONVRDK3RU2PGGIBBYHYKPXLUIW4DH3CM7DHE24Q, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:08.289045904 (59h24m2.720126035s) [noise=true , address=104.36.82.47:4537 , piece_id=4KQCMB7R56S6EKMXLF2E7FWAWPOG3LC2KYHFIKOSVWYRVRESYWAQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:08.403323457 (59h24m2.834403588s) [noise=true , address=104.36.82.59:4321 , piece_id=H7R47I3JTPUGN6FZC3RWZQYUACTALXGAY5YLSEHNHRFMF5WPZNCA, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:08.448769790 (59h24m2.879849921s) [noise=true , address=104.36.82.8:4978 , piece_id=JFT7KYLTNEP3WVBHGY52MWSLXVFP47EWVTQCFEEXO3VTMIMDLJPQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:08.729726419 (59h24m3.16080655s) [noise=true , address=104.36.82.14:4247 , piece_id=N4AEYFHDVO2BO2ZHBZ3MLCTP7OT7WK5QQU7FRAANSMFGWZHXM3DA, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:08.838948988 (59h24m3.270029109s) [noise=true , address=104.36.82.62:4419 , piece_id=JA7GHSJA24CQS2WDSD6V4Z6HZ6G5Y64R7UHJEZJBQLNBTEFFQVVQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:08.887527485 (59h24m3.318607626s) [noise=true , address=104.36.82.55:4199 , piece_id=C5HHQYRNV4GJJDETVYB6EQUVQYPLEDOHTNHFJGOJABSJTXQUQRAA, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:08.914933422 (59h24m3.346013544s) [noise=true , address=104.36.82.12:4167 , piece_id=MGBXFYX5DYMVKEWLKWUASKICDFTG23M26ZCODE4F7IQFG2UVQH6Q, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:09.204636666 (59h24m3.635716797s) [noise=true , address=104.36.82.13:4197 , piece_id=NWUXYJSX2CWINMKSEWPJPESDCDJDNK6RG7DOSXSVROZHAZ4LFR3Q, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:09.204663507 (59h24m3.635743628s) [seg_pos={0 0}, upload=61903]: Segment reached optimal threshold of 14 pieces. Aug 11 04:50:09.204765702 (59h24m3.635845833s) [noise=true , address=104.36.82.16:4305 , piece_id=OZJMNIZFC3P6ZMFPJ2TCC5ARRMSEZ2Z4FAWXGHNVPXJ67RYFC7TQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZMhNJYCNg8K16ZqsVTLBCwGn51vtG5u56cxcqgZPfQaJxgNZv): protocol: expected piece hash; context canceled; EOF Aug 11 04:50:09.204860434 (59h24m3.635940555s) [noise=true , address=104.36.82.23:4453 , piece_id=DDI55E5BTVW3MPNJSOMSA4IGQCOA5WQ2PZHS2VVM2SOLWJMVFGLQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ): protocol: expected piece hash; context canceled; EOF Aug 11 04:50:09.204917162 (59h24m3.635997293s) [noise=true , address=104.36.82.59:4328 , piece_id=UPVR344E5YWFRFQI5LWHPTQFFJU2VBT6V7EFEAB5TNMML5L5WYNQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JXYpLarXKLWgAMdNhxWfTWb9VYWGyaY4yLGtqXAWcCBAYJRZY): protocol: expected piece hash; context canceled; EOF Aug 11 04:50:09.204977537 (59h24m3.636057668s) [noise=true , address=104.36.82.60:4360 , piece_id=EOHB45DR43LHMLLCOUHSWPJ2NUDWJJO3UOHTCYLEVXLNKYAOK4CQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: expected piece hash; context canceled; EOF Aug 11 04:50:09.204992216 (59h24m3.636072347s) [seg_pos={0 0}, upload=61903]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 11 04:50:09.205049335 (59h24m3.636129466s) [upload=61903]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 11 04:50:09.205054926 (59h24m3.636135047s) [upload=61903]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:50:09.504318760 (59h24m3.935398891s) [upload=61903]: Done uploading segments... Aug 11 04:50:09.504336904 (59h24m3.935417035s) [upload=61903]: Upload finished. err: <nil> Duration: 14.613063813s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....X............................................X..........] duration: 1.883937083s FAIL (2025-08-11 04:52:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 04:44:24.932296741 (59h18m19.363376862s) [upload=61797]: Starting upload Aug 11 04:44:24.932375572 (59h18m19.363455703s) [upload=61797]: Done starting upload Aug 11 04:44:24.932485803 (59h18m19.363565994s) [upload=61797]: Uploading segments... Aug 11 04:44:24.932550086 (59h18m19.363630247s) [upload=61797]: Got next segment. Inline: true Aug 11 04:44:24.932557180 (59h18m19.363637331s) [upload=61797]: Waiting for error group managing segments... Aug 11 04:44:24.932610262 (59h18m19.363690453s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:44:24.932616814 (59h18m19.363696995s) [upload=61797]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:44:24.932621844 (59h18m19.363702005s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:44:25.294615282 (59h18m19.725695413s) [upload=61797]: Done uploading segments... Aug 11 04:44:25.294637325 (59h18m19.725717456s) [upload=61797]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.884041553s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....X............................................X.........] duration: 2.042601646s FAIL (2025-08-11 04:51:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 04:44:24.932296741 (59h18m19.363376862s) [upload=61797]: Starting upload Aug 11 04:44:24.932375572 (59h18m19.363455703s) [upload=61797]: Done starting upload Aug 11 04:44:24.932485803 (59h18m19.363565994s) [upload=61797]: Uploading segments... Aug 11 04:44:24.932550086 (59h18m19.363630247s) [upload=61797]: Got next segment. Inline: true Aug 11 04:44:24.932557180 (59h18m19.363637331s) [upload=61797]: Waiting for error group managing segments... Aug 11 04:44:24.932610262 (59h18m19.363690453s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:44:24.932616814 (59h18m19.363696995s) [upload=61797]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:44:24.932621844 (59h18m19.363702005s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:44:25.294615282 (59h18m19.725695413s) [upload=61797]: Done uploading segments... Aug 11 04:44:25.294637325 (59h18m19.725717456s) [upload=61797]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.04264028s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 8.579065687s FAIL (2025-08-11 04:51:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.579090956s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 8.194124149s FAIL (2025-08-11 04:51:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.194151723s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X..............................XX..................X.....X.] duration: 7.393220872s FAIL (2025-08-11 04:51:01 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 11 04:50:04.654122771 (59h23m59.085202902s) [upload=61903]: Starting upload Aug 11 04:50:04.654167006 (59h23m59.085247137s) [upload=61903]: Done starting upload Aug 11 04:50:04.654293087 (59h23m59.085373218s) [upload=61903]: Uploading segments... Aug 11 04:50:04.654458323 (59h23m59.085538454s) [upload=61903]: Got next segment. Inline: false Aug 11 04:50:04.654484242 (59h23m59.085564373s) [upload=61903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:50:04.654489162 (59h23m59.085569293s) [upload=61903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:50:05.098864103 (59h23m59.529944254s) [seg_pos={0 0}, upload=61903]: Begin upload segment... Aug 11 04:50:05.098932725 (59h23m59.530012866s) [seg_pos={0 0}, upload=61903]: Done begin upload segment. Aug 11 04:50:05.098964245 (59h23m59.530044386s) [noise=true , address=104.36.82.47:4537 , piece_id=4KQCMB7R56S6EKMXLF2E7FWAWPOG3LC2KYHFIKOSVWYRVRESYWAQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.099043326 (59h23m59.530123487s) [noise=true , address=104.36.82.59:4328 , piece_id=UPVR344E5YWFRFQI5LWHPTQFFJU2VBT6V7EFEAB5TNMML5L5WYNQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.099548633 (59h23m59.530628814s) [noise=true , address=104.36.82.39:4755 , piece_id=AWVSJKMX5FP4NURUUGJ3S7MLQM2L67AYEFAERTHZ7RCXZC7ORMAQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.099872773 (59h23m59.530952904s) [noise=true , address=104.36.82.46:4487 , piece_id=7AKA2FLAKZKFGRAWVMLUBCDGLEVSKVAWJZY4EVKZKII4JQJWI7IQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.099991821 (59h23m59.531071952s) [noise=true , address=104.36.82.59:4321 , piece_id=H7R47I3JTPUGN6FZC3RWZQYUACTALXGAY5YLSEHNHRFMF5WPZNCA, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.100415351 (59h23m59.531495492s) [noise=true , address=104.36.82.23:4453 , piece_id=DDI55E5BTVW3MPNJSOMSA4IGQCOA5WQ2PZHS2VVM2SOLWJMVFGLQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.100426312 (59h23m59.531506443s) [noise=true , address=104.36.82.60:4360 , piece_id=EOHB45DR43LHMLLCOUHSWPJ2NUDWJJO3UOHTCYLEVXLNKYAOK4CQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.100818353 (59h23m59.531898494s) [noise=true , address=104.36.82.12:4167 , piece_id=MGBXFYX5DYMVKEWLKWUASKICDFTG23M26ZCODE4F7IQFG2UVQH6Q, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.101245610 (59h23m59.532325751s) [noise=true , address=104.36.82.8:4978 , piece_id=JFT7KYLTNEP3WVBHGY52MWSLXVFP47EWVTQCFEEXO3VTMIMDLJPQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.101600088 (59h23m59.532680219s) [noise=true , address=104.36.82.37:4666 , piece_id=QGHN6WDXGXHE5ONVRDK3RU2PGGIBBYHYKPXLUIW4DH3CM7DHE24Q, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.102050571 (59h23m59.533130712s) [noise=true , address=104.36.82.55:4199 , piece_id=C5HHQYRNV4GJJDETVYB6EQUVQYPLEDOHTNHFJGOJABSJTXQUQRAA, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.102458752 (59h23m59.533538883s) [noise=true , address=104.36.82.58:4282 , piece_id=5KOWL5AF4VBFWRHLFCR3HPZT4O3FZ2N2XBFOSU6UVFT54GL3S34A, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.102775637 (59h23m59.533855768s) [noise=true , address=104.36.82.45:4907 , piece_id=I4QDBJDKUIWFAD7BKMWFJSD75I24R2DLKORKCUQ6P7BQVLVLGL2A, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.103089639 (59h23m59.53416977s) [noise=true , address=64.49.13.27:4961 , piece_id=4Q6BHP3IMIMSBP6XDATZCNUUVINRH4N7AKRCTCSOAQVJY4ZGCLIQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.103547073 (59h23m59.534627214s) [noise=true , address=104.36.82.16:4305 , piece_id=OZJMNIZFC3P6ZMFPJ2TCC5ARRMSEZ2Z4FAWXGHNVPXJ67RYFC7TQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.103718020 (59h23m59.534798161s) [noise=true , address=104.36.82.62:4419 , piece_id=JA7GHSJA24CQS2WDSD6V4Z6HZ6G5Y64R7UHJEZJBQLNBTEFFQVVQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.104095302 (59h23m59.535175433s) [noise=true , address=104.36.82.14:4247 , piece_id=N4AEYFHDVO2BO2ZHBZ3MLCTP7OT7WK5QQU7FRAANSMFGWZHXM3DA, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.104403903 (59h23m59.535484044s) [noise=true , address=104.36.82.13:4197 , piece_id=NWUXYJSX2CWINMKSEWPJPESDCDJDNK6RG7DOSXSVROZHAZ4LFR3Q, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:06.295215072 (59h24m0.726295203s) [upload=61903]: Next returned nil segment Aug 11 04:50:06.295232645 (59h24m0.726312776s) [upload=61903]: Waiting for error group managing segments... Aug 11 04:50:06.691486175 (59h24m1.122566296s) [noise=true , address=64.49.13.27:4961 , piece_id=4Q6BHP3IMIMSBP6XDATZCNUUVINRH4N7AKRCTCSOAQVJY4ZGCLIQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:07.192517489 (59h24m1.62359762s) [noise=true , address=104.36.82.45:4907 , piece_id=I4QDBJDKUIWFAD7BKMWFJSD75I24R2DLKORKCUQ6P7BQVLVLGL2A, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:07.301483609 (59h24m1.73256373s) [noise=true , address=104.36.82.58:4282 , piece_id=5KOWL5AF4VBFWRHLFCR3HPZT4O3FZ2N2XBFOSU6UVFT54GL3S34A, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:07.515352383 (59h24m1.946432504s) [noise=true , address=104.36.82.46:4487 , piece_id=7AKA2FLAKZKFGRAWVMLUBCDGLEVSKVAWJZY4EVKZKII4JQJWI7IQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:07.774866883 (59h24m2.205947014s) [noise=true , address=104.36.82.39:4755 , piece_id=AWVSJKMX5FP4NURUUGJ3S7MLQM2L67AYEFAERTHZ7RCXZC7ORMAQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:08.284029245 (59h24m2.715109366s) [noise=true , address=104.36.82.37:4666 , piece_id=QGHN6WDXGXHE5ONVRDK3RU2PGGIBBYHYKPXLUIW4DH3CM7DHE24Q, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:08.289045904 (59h24m2.720126035s) [noise=true , address=104.36.82.47:4537 , piece_id=4KQCMB7R56S6EKMXLF2E7FWAWPOG3LC2KYHFIKOSVWYRVRESYWAQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:08.403323457 (59h24m2.834403588s) [noise=true , address=104.36.82.59:4321 , piece_id=H7R47I3JTPUGN6FZC3RWZQYUACTALXGAY5YLSEHNHRFMF5WPZNCA, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:08.448769790 (59h24m2.879849921s) [noise=true , address=104.36.82.8:4978 , piece_id=JFT7KYLTNEP3WVBHGY52MWSLXVFP47EWVTQCFEEXO3VTMIMDLJPQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:08.729726419 (59h24m3.16080655s) [noise=true , address=104.36.82.14:4247 , piece_id=N4AEYFHDVO2BO2ZHBZ3MLCTP7OT7WK5QQU7FRAANSMFGWZHXM3DA, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:08.838948988 (59h24m3.270029109s) [noise=true , address=104.36.82.62:4419 , piece_id=JA7GHSJA24CQS2WDSD6V4Z6HZ6G5Y64R7UHJEZJBQLNBTEFFQVVQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:08.887527485 (59h24m3.318607626s) [noise=true , address=104.36.82.55:4199 , piece_id=C5HHQYRNV4GJJDETVYB6EQUVQYPLEDOHTNHFJGOJABSJTXQUQRAA, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:08.914933422 (59h24m3.346013544s) [noise=true , address=104.36.82.12:4167 , piece_id=MGBXFYX5DYMVKEWLKWUASKICDFTG23M26ZCODE4F7IQFG2UVQH6Q, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:09.204636666 (59h24m3.635716797s) [noise=true , address=104.36.82.13:4197 , piece_id=NWUXYJSX2CWINMKSEWPJPESDCDJDNK6RG7DOSXSVROZHAZ4LFR3Q, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:09.204663507 (59h24m3.635743628s) [seg_pos={0 0}, upload=61903]: Segment reached optimal threshold of 14 pieces. Aug 11 04:50:09.204765702 (59h24m3.635845833s) [noise=true , address=104.36.82.16:4305 , piece_id=OZJMNIZFC3P6ZMFPJ2TCC5ARRMSEZ2Z4FAWXGHNVPXJ67RYFC7TQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZMhNJYCNg8K16ZqsVTLBCwGn51vtG5u56cxcqgZPfQaJxgNZv): protocol: expected piece hash; context canceled; EOF Aug 11 04:50:09.204860434 (59h24m3.635940555s) [noise=true , address=104.36.82.23:4453 , piece_id=DDI55E5BTVW3MPNJSOMSA4IGQCOA5WQ2PZHS2VVM2SOLWJMVFGLQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ): protocol: expected piece hash; context canceled; EOF Aug 11 04:50:09.204917162 (59h24m3.635997293s) [noise=true , address=104.36.82.59:4328 , piece_id=UPVR344E5YWFRFQI5LWHPTQFFJU2VBT6V7EFEAB5TNMML5L5WYNQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JXYpLarXKLWgAMdNhxWfTWb9VYWGyaY4yLGtqXAWcCBAYJRZY): protocol: expected piece hash; context canceled; EOF Aug 11 04:50:09.204977537 (59h24m3.636057668s) [noise=true , address=104.36.82.60:4360 , piece_id=EOHB45DR43LHMLLCOUHSWPJ2NUDWJJO3UOHTCYLEVXLNKYAOK4CQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: expected piece hash; context canceled; EOF Aug 11 04:50:09.204992216 (59h24m3.636072347s) [seg_pos={0 0}, upload=61903]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 11 04:50:09.205049335 (59h24m3.636129466s) [upload=61903]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 11 04:50:09.205054926 (59h24m3.636135047s) [upload=61903]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:50:09.504318760 (59h24m3.935398891s) [upload=61903]: Done uploading segments... Aug 11 04:50:09.504336904 (59h24m3.935417035s) [upload=61903]: Upload finished. err: <nil> Duration: 7.393251619s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......X............................................X........] duration: 1.856612451s FAIL (2025-08-11 04:50:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 04:44:24.932296741 (59h18m19.363376862s) [upload=61797]: Starting upload Aug 11 04:44:24.932375572 (59h18m19.363455703s) [upload=61797]: Done starting upload Aug 11 04:44:24.932485803 (59h18m19.363565994s) [upload=61797]: Uploading segments... Aug 11 04:44:24.932550086 (59h18m19.363630247s) [upload=61797]: Got next segment. Inline: true Aug 11 04:44:24.932557180 (59h18m19.363637331s) [upload=61797]: Waiting for error group managing segments... Aug 11 04:44:24.932610262 (59h18m19.363690453s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:44:24.932616814 (59h18m19.363696995s) [upload=61797]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:44:24.932621844 (59h18m19.363702005s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:44:25.294615282 (59h18m19.725695413s) [upload=61797]: Done uploading segments... Aug 11 04:44:25.294637325 (59h18m19.725717456s) [upload=61797]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.856639484s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 8.179561536s FAIL (2025-08-11 04:50:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.179591864s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X..............................XX..................X.....X] duration: 6.190825393s FAIL (2025-08-11 04:50:10 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 11 04:50:04.654122771 (59h23m59.085202902s) [upload=61903]: Starting upload Aug 11 04:50:04.654167006 (59h23m59.085247137s) [upload=61903]: Done starting upload Aug 11 04:50:04.654293087 (59h23m59.085373218s) [upload=61903]: Uploading segments... Aug 11 04:50:04.654458323 (59h23m59.085538454s) [upload=61903]: Got next segment. Inline: false Aug 11 04:50:04.654484242 (59h23m59.085564373s) [upload=61903]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:50:04.654489162 (59h23m59.085569293s) [upload=61903]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:50:05.098864103 (59h23m59.529944254s) [seg_pos={0 0}, upload=61903]: Begin upload segment... Aug 11 04:50:05.098932725 (59h23m59.530012866s) [seg_pos={0 0}, upload=61903]: Done begin upload segment. Aug 11 04:50:05.098964245 (59h23m59.530044386s) [noise=true , address=104.36.82.47:4537 , piece_id=4KQCMB7R56S6EKMXLF2E7FWAWPOG3LC2KYHFIKOSVWYRVRESYWAQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.099043326 (59h23m59.530123487s) [noise=true , address=104.36.82.59:4328 , piece_id=UPVR344E5YWFRFQI5LWHPTQFFJU2VBT6V7EFEAB5TNMML5L5WYNQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.099548633 (59h23m59.530628814s) [noise=true , address=104.36.82.39:4755 , piece_id=AWVSJKMX5FP4NURUUGJ3S7MLQM2L67AYEFAERTHZ7RCXZC7ORMAQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.099872773 (59h23m59.530952904s) [noise=true , address=104.36.82.46:4487 , piece_id=7AKA2FLAKZKFGRAWVMLUBCDGLEVSKVAWJZY4EVKZKII4JQJWI7IQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.099991821 (59h23m59.531071952s) [noise=true , address=104.36.82.59:4321 , piece_id=H7R47I3JTPUGN6FZC3RWZQYUACTALXGAY5YLSEHNHRFMF5WPZNCA, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.100415351 (59h23m59.531495492s) [noise=true , address=104.36.82.23:4453 , piece_id=DDI55E5BTVW3MPNJSOMSA4IGQCOA5WQ2PZHS2VVM2SOLWJMVFGLQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.100426312 (59h23m59.531506443s) [noise=true , address=104.36.82.60:4360 , piece_id=EOHB45DR43LHMLLCOUHSWPJ2NUDWJJO3UOHTCYLEVXLNKYAOK4CQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.100818353 (59h23m59.531898494s) [noise=true , address=104.36.82.12:4167 , piece_id=MGBXFYX5DYMVKEWLKWUASKICDFTG23M26ZCODE4F7IQFG2UVQH6Q, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.101245610 (59h23m59.532325751s) [noise=true , address=104.36.82.8:4978 , piece_id=JFT7KYLTNEP3WVBHGY52MWSLXVFP47EWVTQCFEEXO3VTMIMDLJPQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.101600088 (59h23m59.532680219s) [noise=true , address=104.36.82.37:4666 , piece_id=QGHN6WDXGXHE5ONVRDK3RU2PGGIBBYHYKPXLUIW4DH3CM7DHE24Q, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.102050571 (59h23m59.533130712s) [noise=true , address=104.36.82.55:4199 , piece_id=C5HHQYRNV4GJJDETVYB6EQUVQYPLEDOHTNHFJGOJABSJTXQUQRAA, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.102458752 (59h23m59.533538883s) [noise=true , address=104.36.82.58:4282 , piece_id=5KOWL5AF4VBFWRHLFCR3HPZT4O3FZ2N2XBFOSU6UVFT54GL3S34A, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.102775637 (59h23m59.533855768s) [noise=true , address=104.36.82.45:4907 , piece_id=I4QDBJDKUIWFAD7BKMWFJSD75I24R2DLKORKCUQ6P7BQVLVLGL2A, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.103089639 (59h23m59.53416977s) [noise=true , address=64.49.13.27:4961 , piece_id=4Q6BHP3IMIMSBP6XDATZCNUUVINRH4N7AKRCTCSOAQVJY4ZGCLIQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.103547073 (59h23m59.534627214s) [noise=true , address=104.36.82.16:4305 , piece_id=OZJMNIZFC3P6ZMFPJ2TCC5ARRMSEZ2Z4FAWXGHNVPXJ67RYFC7TQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.103718020 (59h23m59.534798161s) [noise=true , address=104.36.82.62:4419 , piece_id=JA7GHSJA24CQS2WDSD6V4Z6HZ6G5Y64R7UHJEZJBQLNBTEFFQVVQ, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.104095302 (59h23m59.535175433s) [noise=true , address=104.36.82.14:4247 , piece_id=N4AEYFHDVO2BO2ZHBZ3MLCTP7OT7WK5QQU7FRAANSMFGWZHXM3DA, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:05.104403903 (59h23m59.535484044s) [noise=true , address=104.36.82.13:4197 , piece_id=NWUXYJSX2CWINMKSEWPJPESDCDJDNK6RG7DOSXSVROZHAZ4LFR3Q, seg_pos={0 0}, upload=61903]: Uploading piece... Aug 11 04:50:06.295215072 (59h24m0.726295203s) [upload=61903]: Next returned nil segment Aug 11 04:50:06.295232645 (59h24m0.726312776s) [upload=61903]: Waiting for error group managing segments... Aug 11 04:50:06.691486175 (59h24m1.122566296s) [noise=true , address=64.49.13.27:4961 , piece_id=4Q6BHP3IMIMSBP6XDATZCNUUVINRH4N7AKRCTCSOAQVJY4ZGCLIQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:07.192517489 (59h24m1.62359762s) [noise=true , address=104.36.82.45:4907 , piece_id=I4QDBJDKUIWFAD7BKMWFJSD75I24R2DLKORKCUQ6P7BQVLVLGL2A, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:07.301483609 (59h24m1.73256373s) [noise=true , address=104.36.82.58:4282 , piece_id=5KOWL5AF4VBFWRHLFCR3HPZT4O3FZ2N2XBFOSU6UVFT54GL3S34A, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:07.515352383 (59h24m1.946432504s) [noise=true , address=104.36.82.46:4487 , piece_id=7AKA2FLAKZKFGRAWVMLUBCDGLEVSKVAWJZY4EVKZKII4JQJWI7IQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:07.774866883 (59h24m2.205947014s) [noise=true , address=104.36.82.39:4755 , piece_id=AWVSJKMX5FP4NURUUGJ3S7MLQM2L67AYEFAERTHZ7RCXZC7ORMAQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:08.284029245 (59h24m2.715109366s) [noise=true , address=104.36.82.37:4666 , piece_id=QGHN6WDXGXHE5ONVRDK3RU2PGGIBBYHYKPXLUIW4DH3CM7DHE24Q, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:08.289045904 (59h24m2.720126035s) [noise=true , address=104.36.82.47:4537 , piece_id=4KQCMB7R56S6EKMXLF2E7FWAWPOG3LC2KYHFIKOSVWYRVRESYWAQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:08.403323457 (59h24m2.834403588s) [noise=true , address=104.36.82.59:4321 , piece_id=H7R47I3JTPUGN6FZC3RWZQYUACTALXGAY5YLSEHNHRFMF5WPZNCA, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:08.448769790 (59h24m2.879849921s) [noise=true , address=104.36.82.8:4978 , piece_id=JFT7KYLTNEP3WVBHGY52MWSLXVFP47EWVTQCFEEXO3VTMIMDLJPQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:08.729726419 (59h24m3.16080655s) [noise=true , address=104.36.82.14:4247 , piece_id=N4AEYFHDVO2BO2ZHBZ3MLCTP7OT7WK5QQU7FRAANSMFGWZHXM3DA, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:08.838948988 (59h24m3.270029109s) [noise=true , address=104.36.82.62:4419 , piece_id=JA7GHSJA24CQS2WDSD6V4Z6HZ6G5Y64R7UHJEZJBQLNBTEFFQVVQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:08.887527485 (59h24m3.318607626s) [noise=true , address=104.36.82.55:4199 , piece_id=C5HHQYRNV4GJJDETVYB6EQUVQYPLEDOHTNHFJGOJABSJTXQUQRAA, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:08.914933422 (59h24m3.346013544s) [noise=true , address=104.36.82.12:4167 , piece_id=MGBXFYX5DYMVKEWLKWUASKICDFTG23M26ZCODE4F7IQFG2UVQH6Q, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:09.204636666 (59h24m3.635716797s) [noise=true , address=104.36.82.13:4197 , piece_id=NWUXYJSX2CWINMKSEWPJPESDCDJDNK6RG7DOSXSVROZHAZ4LFR3Q, seg_pos={0 0}, upload=61903]: Done uploading piece. err: <nil> Aug 11 04:50:09.204663507 (59h24m3.635743628s) [seg_pos={0 0}, upload=61903]: Segment reached optimal threshold of 14 pieces. Aug 11 04:50:09.204765702 (59h24m3.635845833s) [noise=true , address=104.36.82.16:4305 , piece_id=OZJMNIZFC3P6ZMFPJ2TCC5ARRMSEZ2Z4FAWXGHNVPXJ67RYFC7TQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZMhNJYCNg8K16ZqsVTLBCwGn51vtG5u56cxcqgZPfQaJxgNZv): protocol: expected piece hash; context canceled; EOF Aug 11 04:50:09.204860434 (59h24m3.635940555s) [noise=true , address=104.36.82.23:4453 , piece_id=DDI55E5BTVW3MPNJSOMSA4IGQCOA5WQ2PZHS2VVM2SOLWJMVFGLQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ktniPbRibXzsTSNW9MbWWxymt9gZKU8sDgTjixc6nYzT9c5tJ): protocol: expected piece hash; context canceled; EOF Aug 11 04:50:09.204917162 (59h24m3.635997293s) [noise=true , address=104.36.82.59:4328 , piece_id=UPVR344E5YWFRFQI5LWHPTQFFJU2VBT6V7EFEAB5TNMML5L5WYNQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JXYpLarXKLWgAMdNhxWfTWb9VYWGyaY4yLGtqXAWcCBAYJRZY): protocol: expected piece hash; context canceled; EOF Aug 11 04:50:09.204977537 (59h24m3.636057668s) [noise=true , address=104.36.82.60:4360 , piece_id=EOHB45DR43LHMLLCOUHSWPJ2NUDWJJO3UOHTCYLEVXLNKYAOK4CQ, seg_pos={0 0}, upload=61903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: expected piece hash; context canceled; EOF Aug 11 04:50:09.204992216 (59h24m3.636072347s) [seg_pos={0 0}, upload=61903]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 11 04:50:09.205049335 (59h24m3.636129466s) [upload=61903]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 11 04:50:09.205054926 (59h24m3.636135047s) [upload=61903]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:50:09.504318760 (59h24m3.935398891s) [upload=61903]: Done uploading segments... Aug 11 04:50:09.504336904 (59h24m3.935417035s) [upload=61903]: Upload finished. err: <nil> Duration: 6.190859618s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X..............................XX..................X.....] duration: 10.203244961s FAIL (2025-08-11 04:49:39 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 11 04:45:27.852954309 (59h19m22.28403444s) [upload=61818]: Starting upload Aug 11 04:45:27.853020255 (59h19m22.284100386s) [upload=61818]: Done starting upload Aug 11 04:45:27.853161035 (59h19m22.284241166s) [upload=61818]: Uploading segments... Aug 11 04:45:27.853265945 (59h19m22.284346066s) [upload=61818]: Got next segment. Inline: false Aug 11 04:45:27.853313025 (59h19m22.284393156s) [upload=61818]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:45:27.853318035 (59h19m22.284398166s) [upload=61818]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:45:28.311646796 (59h19m22.742726927s) [seg_pos={0 0}, upload=61818]: Begin upload segment... Aug 11 04:45:28.311691702 (59h19m22.742771833s) [seg_pos={0 0}, upload=61818]: Done begin upload segment. Aug 11 04:45:28.311716960 (59h19m22.742797081s) [noise=true , address=104.36.82.35:4586 , piece_id=DKRAQRI6YALBTQ5W5AZPU4D7I6PDLV5NNLHF6NJCZRENBSEYQ5BA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.311742379 (59h19m22.74282251s) [noise=true , address=104.36.82.9:4011 , piece_id=BGLEADLDLGTOYQPJV6CLTEGQFQ5L7M5YHFQL674224IKCVFJ4PKA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312118228 (59h19m22.743198349s) [noise=true , address=104.36.82.37:4683 , piece_id=CN5FO5OYCPARGFIMP2VSUB2FHJGFYI7WGSEHXAEFTCXIAK3QEW5A, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312148716 (59h19m22.743228847s) [noise=true , address=104.36.82.23:4456 , piece_id=RXQYJINMERMKG4QRBUP635ELCEM2SEPHXKBN7V62BBCZJLQVRKUA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312386371 (59h19m22.743466503s) [noise=true , address=104.36.82.38:4711 , piece_id=RQOEDOR73GM4W4KNLGPS542JHIIT7XKOZXQ2YVLQR6JCQUPD23IQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312483447 (59h19m22.743563578s) [noise=true , address=104.36.82.15:4272 , piece_id=RUWJNFZUWFPPCHKIYKLM65NRZ7URCPY33IWVRI6DNR4L2BLRRRSQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312662880 (59h19m22.743743032s) [noise=true , address=104.36.82.16:4322 , piece_id=3JFTHIYCW53BZK6UPHAQSALUXC45N3IMPDYVCMA4NJN7C57QIOBQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312840229 (59h19m22.74392036s) [noise=true , address=104.36.82.2:4734 , piece_id=3GU3VG5HCSTKNLP2L4FHMSWMVQYEVTP3SKAT3WRFCNM7IGZ5XUAA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312973755 (59h19m22.744053886s) [noise=true , address=64.49.13.154:4930 , piece_id=CXEIO7OJ75OUXS5RD666FYX5CBP32HMTEJK2TOPFGE7K24Z3HBBQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313147277 (59h19m22.744227398s) [noise=true , address=104.36.82.58:4265 , piece_id=JYMYFO4ZRS7GU3S3YUG2STDZAORUK52TDX74FC2XOETWILKI5NRA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313215117 (59h19m22.744295248s) [noise=true , address=77.90.31.16:4741 , piece_id=ROWNQREQDOXSZ33OCVPUTHCWKKFQ2UJNGA3O7VHLPYNNLLCP7ALQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313409038 (59h19m22.744489169s) [noise=true , address=104.36.82.60:4358 , piece_id=LMZE7L5QDPL7KO6E7S5LTXKVRRZ2MKLJHJ5YYGUXMAFLXXIDRZMQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313454284 (59h19m22.744534415s) [noise=true , address=104.36.82.35:4588 , piece_id=SHILYCT7SE26CH26FCFWULB2I5IF5QTBOZXDPGR4YE2X4WFQ5YPQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313693011 (59h19m22.744773132s) [noise=true , address=104.36.82.62:4427 , piece_id=JA6FYXOYVOW2ABSKW6OSXF52O5ADF34NCELQJMF5MXWY5KURF5OQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313755310 (59h19m22.744835441s) [noise=true , address=104.36.82.5:4849 , piece_id=CRQSKVDPYQ2PW52COXWMQWWWWJAMWW2A2SSLNEQWFIASJMYX2PPQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313910657 (59h19m22.744990789s) [noise=true , address=104.36.82.38:4721 , piece_id=ABD3F372AYA2CSGX6YIFF4455YEKUA4MCSFH6P4LT3UWQMITWMXQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.314072688 (59h19m22.745152819s) [noise=true , address=45.91.116.11:4136 , piece_id=OXZMVFV6D7I622HOU5DMS7TW5FNJRVCEW4TM7CVCMM62VC6J7FVQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.314170785 (59h19m22.745250916s) [noise=true , address=104.36.82.59:4310 , piece_id=EIKGAUOEAG246IR632UUJSEYPYCTQ4JTFR435XAWJVH6ERN5KS2A, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:29.543198877 (59h19m23.974279008s) [upload=61818]: Next returned nil segment Aug 11 04:45:29.543221601 (59h19m23.974301732s) [upload=61818]: Waiting for error group managing segments... Aug 11 04:45:29.949775487 (59h19m24.380855618s) [noise=true , address=64.49.13.154:4930 , piece_id=CXEIO7OJ75OUXS5RD666FYX5CBP32HMTEJK2TOPFGE7K24Z3HBBQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.527562944 (59h19m24.958643075s) [noise=true , address=104.36.82.9:4011 , piece_id=BGLEADLDLGTOYQPJV6CLTEGQFQ5L7M5YHFQL674224IKCVFJ4PKA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.672823547 (59h19m25.103903678s) [noise=true , address=104.36.82.59:4310 , piece_id=EIKGAUOEAG246IR632UUJSEYPYCTQ4JTFR435XAWJVH6ERN5KS2A, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.695652536 (59h19m25.126732667s) [noise=true , address=104.36.82.38:4711 , piece_id=RQOEDOR73GM4W4KNLGPS542JHIIT7XKOZXQ2YVLQR6JCQUPD23IQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.868012671 (59h19m25.299092802s) [noise=true , address=104.36.82.37:4683 , piece_id=CN5FO5OYCPARGFIMP2VSUB2FHJGFYI7WGSEHXAEFTCXIAK3QEW5A, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.900922148 (59h19m25.332002279s) [noise=true , address=104.36.82.2:4734 , piece_id=3GU3VG5HCSTKNLP2L4FHMSWMVQYEVTP3SKAT3WRFCNM7IGZ5XUAA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.935177250 (59h19m25.366257381s) [noise=true , address=104.36.82.35:4586 , piece_id=DKRAQRI6YALBTQ5W5AZPU4D7I6PDLV5NNLHF6NJCZRENBSEYQ5BA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.966459844 (59h19m25.397539975s) [noise=true , address=104.36.82.58:4265 , piece_id=JYMYFO4ZRS7GU3S3YUG2STDZAORUK52TDX74FC2XOETWILKI5NRA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.050105808 (59h19m25.481185939s) [noise=true , address=104.36.82.60:4358 , piece_id=LMZE7L5QDPL7KO6E7S5LTXKVRRZ2MKLJHJ5YYGUXMAFLXXIDRZMQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.059799375 (59h19m25.490879506s) [noise=true , address=104.36.82.62:4427 , piece_id=JA6FYXOYVOW2ABSKW6OSXF52O5ADF34NCELQJMF5MXWY5KURF5OQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.270855508 (59h19m25.701935639s) [noise=true , address=77.90.31.16:4741 , piece_id=ROWNQREQDOXSZ33OCVPUTHCWKKFQ2UJNGA3O7VHLPYNNLLCP7ALQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.363852803 (59h19m25.794932934s) [noise=true , address=104.36.82.23:4456 , piece_id=RXQYJINMERMKG4QRBUP635ELCEM2SEPHXKBN7V62BBCZJLQVRKUA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.386074781 (59h19m25.817154912s) [noise=true , address=104.36.82.38:4721 , piece_id=ABD3F372AYA2CSGX6YIFF4455YEKUA4MCSFH6P4LT3UWQMITWMXQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.425040486 (59h19m25.856120617s) [noise=true , address=45.91.116.11:4136 , piece_id=OXZMVFV6D7I622HOU5DMS7TW5FNJRVCEW4TM7CVCMM62VC6J7FVQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.425075023 (59h19m25.856155154s) [seg_pos={0 0}, upload=61818]: Segment reached optimal threshold of 14 pieces. Aug 11 04:45:31.425249597 (59h19m25.856329758s) [noise=true , address=104.36.82.16:4322 , piece_id=3JFTHIYCW53BZK6UPHAQSALUXC45N3IMPDYVCMA4NJN7C57QIOBQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3): protocol: expected piece hash; context canceled; EOF Aug 11 04:45:31.425348236 (59h19m25.856428367s) [noise=true , address=104.36.82.5:4849 , piece_id=CRQSKVDPYQ2PW52COXWMQWWWWJAMWW2A2SSLNEQWFIASJMYX2PPQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2gSzVmNCEZcfVsFKibjXHs2jMr82D3SWQEykFQQ4uyhhnrj1): protocol: expected piece hash; context canceled; EOF Aug 11 04:45:31.425394103 (59h19m25.856474234s) [noise=true , address=104.36.82.35:4588 , piece_id=SHILYCT7SE26CH26FCFWULB2I5IF5QTBOZXDPGR4YE2X4WFQ5YPQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Aug 11 04:45:31.425471601 (59h19m25.856551742s) [noise=true , address=104.36.82.15:4272 , piece_id=RUWJNFZUWFPPCHKIYKLM65NRZ7URCPY33IWVRI6DNR4L2BLRRRSQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UeKsX3pQfhvCukBvJf1sekK7N6sXkaWaJnZzETKoxyXWo2BFi): protocol: expected piece hash; context canceled; EOF Aug 11 04:45:31.425497581 (59h19m25.856577772s) [seg_pos={0 0}, upload=61818]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 11 04:45:31.425586341 (59h19m25.856666482s) [upload=61818]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 11 04:45:31.425593415 (59h19m25.856673596s) [upload=61818]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:45:31.735520612 (59h19m26.166600743s) [upload=61818]: Done uploading segments... Aug 11 04:45:31.735552864 (59h19m26.166632995s) [upload=61818]: Upload finished. err: <nil> Duration: 10.2032703s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......X............................................X.......] duration: 2.01626612s FAIL (2025-08-11 04:49:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 04:44:24.932296741 (59h18m19.363376862s) [upload=61797]: Starting upload Aug 11 04:44:24.932375572 (59h18m19.363455703s) [upload=61797]: Done starting upload Aug 11 04:44:24.932485803 (59h18m19.363565994s) [upload=61797]: Uploading segments... Aug 11 04:44:24.932550086 (59h18m19.363630247s) [upload=61797]: Got next segment. Inline: true Aug 11 04:44:24.932557180 (59h18m19.363637331s) [upload=61797]: Waiting for error group managing segments... Aug 11 04:44:24.932610262 (59h18m19.363690453s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:44:24.932616814 (59h18m19.363696995s) [upload=61797]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:44:24.932621844 (59h18m19.363702005s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:44:25.294615282 (59h18m19.725695413s) [upload=61797]: Done uploading segments... Aug 11 04:44:25.294637325 (59h18m19.725717456s) [upload=61797]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.01629223s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 8.865946935s FAIL (2025-08-11 04:49:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.865975861s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 9.565729646s FAIL (2025-08-11 04:48:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.565753923s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X..............................XX..................X....] duration: 6.908961861s FAIL (2025-08-11 04:48:28 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 11 04:45:27.852954309 (59h19m22.28403444s) [upload=61818]: Starting upload Aug 11 04:45:27.853020255 (59h19m22.284100386s) [upload=61818]: Done starting upload Aug 11 04:45:27.853161035 (59h19m22.284241166s) [upload=61818]: Uploading segments... Aug 11 04:45:27.853265945 (59h19m22.284346066s) [upload=61818]: Got next segment. Inline: false Aug 11 04:45:27.853313025 (59h19m22.284393156s) [upload=61818]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:45:27.853318035 (59h19m22.284398166s) [upload=61818]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:45:28.311646796 (59h19m22.742726927s) [seg_pos={0 0}, upload=61818]: Begin upload segment... Aug 11 04:45:28.311691702 (59h19m22.742771833s) [seg_pos={0 0}, upload=61818]: Done begin upload segment. Aug 11 04:45:28.311716960 (59h19m22.742797081s) [noise=true , address=104.36.82.35:4586 , piece_id=DKRAQRI6YALBTQ5W5AZPU4D7I6PDLV5NNLHF6NJCZRENBSEYQ5BA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.311742379 (59h19m22.74282251s) [noise=true , address=104.36.82.9:4011 , piece_id=BGLEADLDLGTOYQPJV6CLTEGQFQ5L7M5YHFQL674224IKCVFJ4PKA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312118228 (59h19m22.743198349s) [noise=true , address=104.36.82.37:4683 , piece_id=CN5FO5OYCPARGFIMP2VSUB2FHJGFYI7WGSEHXAEFTCXIAK3QEW5A, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312148716 (59h19m22.743228847s) [noise=true , address=104.36.82.23:4456 , piece_id=RXQYJINMERMKG4QRBUP635ELCEM2SEPHXKBN7V62BBCZJLQVRKUA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312386371 (59h19m22.743466503s) [noise=true , address=104.36.82.38:4711 , piece_id=RQOEDOR73GM4W4KNLGPS542JHIIT7XKOZXQ2YVLQR6JCQUPD23IQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312483447 (59h19m22.743563578s) [noise=true , address=104.36.82.15:4272 , piece_id=RUWJNFZUWFPPCHKIYKLM65NRZ7URCPY33IWVRI6DNR4L2BLRRRSQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312662880 (59h19m22.743743032s) [noise=true , address=104.36.82.16:4322 , piece_id=3JFTHIYCW53BZK6UPHAQSALUXC45N3IMPDYVCMA4NJN7C57QIOBQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312840229 (59h19m22.74392036s) [noise=true , address=104.36.82.2:4734 , piece_id=3GU3VG5HCSTKNLP2L4FHMSWMVQYEVTP3SKAT3WRFCNM7IGZ5XUAA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312973755 (59h19m22.744053886s) [noise=true , address=64.49.13.154:4930 , piece_id=CXEIO7OJ75OUXS5RD666FYX5CBP32HMTEJK2TOPFGE7K24Z3HBBQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313147277 (59h19m22.744227398s) [noise=true , address=104.36.82.58:4265 , piece_id=JYMYFO4ZRS7GU3S3YUG2STDZAORUK52TDX74FC2XOETWILKI5NRA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313215117 (59h19m22.744295248s) [noise=true , address=77.90.31.16:4741 , piece_id=ROWNQREQDOXSZ33OCVPUTHCWKKFQ2UJNGA3O7VHLPYNNLLCP7ALQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313409038 (59h19m22.744489169s) [noise=true , address=104.36.82.60:4358 , piece_id=LMZE7L5QDPL7KO6E7S5LTXKVRRZ2MKLJHJ5YYGUXMAFLXXIDRZMQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313454284 (59h19m22.744534415s) [noise=true , address=104.36.82.35:4588 , piece_id=SHILYCT7SE26CH26FCFWULB2I5IF5QTBOZXDPGR4YE2X4WFQ5YPQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313693011 (59h19m22.744773132s) [noise=true , address=104.36.82.62:4427 , piece_id=JA6FYXOYVOW2ABSKW6OSXF52O5ADF34NCELQJMF5MXWY5KURF5OQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313755310 (59h19m22.744835441s) [noise=true , address=104.36.82.5:4849 , piece_id=CRQSKVDPYQ2PW52COXWMQWWWWJAMWW2A2SSLNEQWFIASJMYX2PPQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313910657 (59h19m22.744990789s) [noise=true , address=104.36.82.38:4721 , piece_id=ABD3F372AYA2CSGX6YIFF4455YEKUA4MCSFH6P4LT3UWQMITWMXQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.314072688 (59h19m22.745152819s) [noise=true , address=45.91.116.11:4136 , piece_id=OXZMVFV6D7I622HOU5DMS7TW5FNJRVCEW4TM7CVCMM62VC6J7FVQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.314170785 (59h19m22.745250916s) [noise=true , address=104.36.82.59:4310 , piece_id=EIKGAUOEAG246IR632UUJSEYPYCTQ4JTFR435XAWJVH6ERN5KS2A, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:29.543198877 (59h19m23.974279008s) [upload=61818]: Next returned nil segment Aug 11 04:45:29.543221601 (59h19m23.974301732s) [upload=61818]: Waiting for error group managing segments... Aug 11 04:45:29.949775487 (59h19m24.380855618s) [noise=true , address=64.49.13.154:4930 , piece_id=CXEIO7OJ75OUXS5RD666FYX5CBP32HMTEJK2TOPFGE7K24Z3HBBQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.527562944 (59h19m24.958643075s) [noise=true , address=104.36.82.9:4011 , piece_id=BGLEADLDLGTOYQPJV6CLTEGQFQ5L7M5YHFQL674224IKCVFJ4PKA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.672823547 (59h19m25.103903678s) [noise=true , address=104.36.82.59:4310 , piece_id=EIKGAUOEAG246IR632UUJSEYPYCTQ4JTFR435XAWJVH6ERN5KS2A, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.695652536 (59h19m25.126732667s) [noise=true , address=104.36.82.38:4711 , piece_id=RQOEDOR73GM4W4KNLGPS542JHIIT7XKOZXQ2YVLQR6JCQUPD23IQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.868012671 (59h19m25.299092802s) [noise=true , address=104.36.82.37:4683 , piece_id=CN5FO5OYCPARGFIMP2VSUB2FHJGFYI7WGSEHXAEFTCXIAK3QEW5A, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.900922148 (59h19m25.332002279s) [noise=true , address=104.36.82.2:4734 , piece_id=3GU3VG5HCSTKNLP2L4FHMSWMVQYEVTP3SKAT3WRFCNM7IGZ5XUAA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.935177250 (59h19m25.366257381s) [noise=true , address=104.36.82.35:4586 , piece_id=DKRAQRI6YALBTQ5W5AZPU4D7I6PDLV5NNLHF6NJCZRENBSEYQ5BA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.966459844 (59h19m25.397539975s) [noise=true , address=104.36.82.58:4265 , piece_id=JYMYFO4ZRS7GU3S3YUG2STDZAORUK52TDX74FC2XOETWILKI5NRA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.050105808 (59h19m25.481185939s) [noise=true , address=104.36.82.60:4358 , piece_id=LMZE7L5QDPL7KO6E7S5LTXKVRRZ2MKLJHJ5YYGUXMAFLXXIDRZMQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.059799375 (59h19m25.490879506s) [noise=true , address=104.36.82.62:4427 , piece_id=JA6FYXOYVOW2ABSKW6OSXF52O5ADF34NCELQJMF5MXWY5KURF5OQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.270855508 (59h19m25.701935639s) [noise=true , address=77.90.31.16:4741 , piece_id=ROWNQREQDOXSZ33OCVPUTHCWKKFQ2UJNGA3O7VHLPYNNLLCP7ALQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.363852803 (59h19m25.794932934s) [noise=true , address=104.36.82.23:4456 , piece_id=RXQYJINMERMKG4QRBUP635ELCEM2SEPHXKBN7V62BBCZJLQVRKUA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.386074781 (59h19m25.817154912s) [noise=true , address=104.36.82.38:4721 , piece_id=ABD3F372AYA2CSGX6YIFF4455YEKUA4MCSFH6P4LT3UWQMITWMXQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.425040486 (59h19m25.856120617s) [noise=true , address=45.91.116.11:4136 , piece_id=OXZMVFV6D7I622HOU5DMS7TW5FNJRVCEW4TM7CVCMM62VC6J7FVQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.425075023 (59h19m25.856155154s) [seg_pos={0 0}, upload=61818]: Segment reached optimal threshold of 14 pieces. Aug 11 04:45:31.425249597 (59h19m25.856329758s) [noise=true , address=104.36.82.16:4322 , piece_id=3JFTHIYCW53BZK6UPHAQSALUXC45N3IMPDYVCMA4NJN7C57QIOBQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3): protocol: expected piece hash; context canceled; EOF Aug 11 04:45:31.425348236 (59h19m25.856428367s) [noise=true , address=104.36.82.5:4849 , piece_id=CRQSKVDPYQ2PW52COXWMQWWWWJAMWW2A2SSLNEQWFIASJMYX2PPQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2gSzVmNCEZcfVsFKibjXHs2jMr82D3SWQEykFQQ4uyhhnrj1): protocol: expected piece hash; context canceled; EOF Aug 11 04:45:31.425394103 (59h19m25.856474234s) [noise=true , address=104.36.82.35:4588 , piece_id=SHILYCT7SE26CH26FCFWULB2I5IF5QTBOZXDPGR4YE2X4WFQ5YPQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Aug 11 04:45:31.425471601 (59h19m25.856551742s) [noise=true , address=104.36.82.15:4272 , piece_id=RUWJNFZUWFPPCHKIYKLM65NRZ7URCPY33IWVRI6DNR4L2BLRRRSQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UeKsX3pQfhvCukBvJf1sekK7N6sXkaWaJnZzETKoxyXWo2BFi): protocol: expected piece hash; context canceled; EOF Aug 11 04:45:31.425497581 (59h19m25.856577772s) [seg_pos={0 0}, upload=61818]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 11 04:45:31.425586341 (59h19m25.856666482s) [upload=61818]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 11 04:45:31.425593415 (59h19m25.856673596s) [upload=61818]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:45:31.735520612 (59h19m26.166600743s) [upload=61818]: Done uploading segments... Aug 11 04:45:31.735552864 (59h19m26.166632995s) [upload=61818]: Upload finished. err: <nil> Duration: 6.908995236s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........X............................................X......] duration: 1.863676683s FAIL (2025-08-11 04:48:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 04:44:24.932296741 (59h18m19.363376862s) [upload=61797]: Starting upload Aug 11 04:44:24.932375572 (59h18m19.363455703s) [upload=61797]: Done starting upload Aug 11 04:44:24.932485803 (59h18m19.363565994s) [upload=61797]: Uploading segments... Aug 11 04:44:24.932550086 (59h18m19.363630247s) [upload=61797]: Got next segment. Inline: true Aug 11 04:44:24.932557180 (59h18m19.363637331s) [upload=61797]: Waiting for error group managing segments... Aug 11 04:44:24.932610262 (59h18m19.363690453s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:44:24.932616814 (59h18m19.363696995s) [upload=61797]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:44:24.932621844 (59h18m19.363702005s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:44:25.294615282 (59h18m19.725695413s) [upload=61797]: Done uploading segments... Aug 11 04:44:25.294637325 (59h18m19.725717456s) [upload=61797]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.863699355s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........X............................................X.....] duration: 1.813089724s FAIL (2025-08-11 04:47:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 04:44:24.932296741 (59h18m19.363376862s) [upload=61797]: Starting upload Aug 11 04:44:24.932375572 (59h18m19.363455703s) [upload=61797]: Done starting upload Aug 11 04:44:24.932485803 (59h18m19.363565994s) [upload=61797]: Uploading segments... Aug 11 04:44:24.932550086 (59h18m19.363630247s) [upload=61797]: Got next segment. Inline: true Aug 11 04:44:24.932557180 (59h18m19.363637331s) [upload=61797]: Waiting for error group managing segments... Aug 11 04:44:24.932610262 (59h18m19.363690453s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:44:24.932616814 (59h18m19.363696995s) [upload=61797]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:44:24.932621844 (59h18m19.363702005s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:44:25.294615282 (59h18m19.725695413s) [upload=61797]: Done uploading segments... Aug 11 04:44:25.294637325 (59h18m19.725717456s) [upload=61797]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.813117807s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 9.132138313s FAIL (2025-08-11 04:47:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.132162671s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X..............................XX..................X...] duration: 9.553871927s FAIL (2025-08-11 04:47:36 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 11 04:45:27.852954309 (59h19m22.28403444s) [upload=61818]: Starting upload Aug 11 04:45:27.853020255 (59h19m22.284100386s) [upload=61818]: Done starting upload Aug 11 04:45:27.853161035 (59h19m22.284241166s) [upload=61818]: Uploading segments... Aug 11 04:45:27.853265945 (59h19m22.284346066s) [upload=61818]: Got next segment. Inline: false Aug 11 04:45:27.853313025 (59h19m22.284393156s) [upload=61818]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:45:27.853318035 (59h19m22.284398166s) [upload=61818]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:45:28.311646796 (59h19m22.742726927s) [seg_pos={0 0}, upload=61818]: Begin upload segment... Aug 11 04:45:28.311691702 (59h19m22.742771833s) [seg_pos={0 0}, upload=61818]: Done begin upload segment. Aug 11 04:45:28.311716960 (59h19m22.742797081s) [noise=true , address=104.36.82.35:4586 , piece_id=DKRAQRI6YALBTQ5W5AZPU4D7I6PDLV5NNLHF6NJCZRENBSEYQ5BA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.311742379 (59h19m22.74282251s) [noise=true , address=104.36.82.9:4011 , piece_id=BGLEADLDLGTOYQPJV6CLTEGQFQ5L7M5YHFQL674224IKCVFJ4PKA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312118228 (59h19m22.743198349s) [noise=true , address=104.36.82.37:4683 , piece_id=CN5FO5OYCPARGFIMP2VSUB2FHJGFYI7WGSEHXAEFTCXIAK3QEW5A, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312148716 (59h19m22.743228847s) [noise=true , address=104.36.82.23:4456 , piece_id=RXQYJINMERMKG4QRBUP635ELCEM2SEPHXKBN7V62BBCZJLQVRKUA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312386371 (59h19m22.743466503s) [noise=true , address=104.36.82.38:4711 , piece_id=RQOEDOR73GM4W4KNLGPS542JHIIT7XKOZXQ2YVLQR6JCQUPD23IQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312483447 (59h19m22.743563578s) [noise=true , address=104.36.82.15:4272 , piece_id=RUWJNFZUWFPPCHKIYKLM65NRZ7URCPY33IWVRI6DNR4L2BLRRRSQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312662880 (59h19m22.743743032s) [noise=true , address=104.36.82.16:4322 , piece_id=3JFTHIYCW53BZK6UPHAQSALUXC45N3IMPDYVCMA4NJN7C57QIOBQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312840229 (59h19m22.74392036s) [noise=true , address=104.36.82.2:4734 , piece_id=3GU3VG5HCSTKNLP2L4FHMSWMVQYEVTP3SKAT3WRFCNM7IGZ5XUAA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312973755 (59h19m22.744053886s) [noise=true , address=64.49.13.154:4930 , piece_id=CXEIO7OJ75OUXS5RD666FYX5CBP32HMTEJK2TOPFGE7K24Z3HBBQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313147277 (59h19m22.744227398s) [noise=true , address=104.36.82.58:4265 , piece_id=JYMYFO4ZRS7GU3S3YUG2STDZAORUK52TDX74FC2XOETWILKI5NRA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313215117 (59h19m22.744295248s) [noise=true , address=77.90.31.16:4741 , piece_id=ROWNQREQDOXSZ33OCVPUTHCWKKFQ2UJNGA3O7VHLPYNNLLCP7ALQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313409038 (59h19m22.744489169s) [noise=true , address=104.36.82.60:4358 , piece_id=LMZE7L5QDPL7KO6E7S5LTXKVRRZ2MKLJHJ5YYGUXMAFLXXIDRZMQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313454284 (59h19m22.744534415s) [noise=true , address=104.36.82.35:4588 , piece_id=SHILYCT7SE26CH26FCFWULB2I5IF5QTBOZXDPGR4YE2X4WFQ5YPQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313693011 (59h19m22.744773132s) [noise=true , address=104.36.82.62:4427 , piece_id=JA6FYXOYVOW2ABSKW6OSXF52O5ADF34NCELQJMF5MXWY5KURF5OQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313755310 (59h19m22.744835441s) [noise=true , address=104.36.82.5:4849 , piece_id=CRQSKVDPYQ2PW52COXWMQWWWWJAMWW2A2SSLNEQWFIASJMYX2PPQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313910657 (59h19m22.744990789s) [noise=true , address=104.36.82.38:4721 , piece_id=ABD3F372AYA2CSGX6YIFF4455YEKUA4MCSFH6P4LT3UWQMITWMXQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.314072688 (59h19m22.745152819s) [noise=true , address=45.91.116.11:4136 , piece_id=OXZMVFV6D7I622HOU5DMS7TW5FNJRVCEW4TM7CVCMM62VC6J7FVQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.314170785 (59h19m22.745250916s) [noise=true , address=104.36.82.59:4310 , piece_id=EIKGAUOEAG246IR632UUJSEYPYCTQ4JTFR435XAWJVH6ERN5KS2A, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:29.543198877 (59h19m23.974279008s) [upload=61818]: Next returned nil segment Aug 11 04:45:29.543221601 (59h19m23.974301732s) [upload=61818]: Waiting for error group managing segments... Aug 11 04:45:29.949775487 (59h19m24.380855618s) [noise=true , address=64.49.13.154:4930 , piece_id=CXEIO7OJ75OUXS5RD666FYX5CBP32HMTEJK2TOPFGE7K24Z3HBBQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.527562944 (59h19m24.958643075s) [noise=true , address=104.36.82.9:4011 , piece_id=BGLEADLDLGTOYQPJV6CLTEGQFQ5L7M5YHFQL674224IKCVFJ4PKA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.672823547 (59h19m25.103903678s) [noise=true , address=104.36.82.59:4310 , piece_id=EIKGAUOEAG246IR632UUJSEYPYCTQ4JTFR435XAWJVH6ERN5KS2A, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.695652536 (59h19m25.126732667s) [noise=true , address=104.36.82.38:4711 , piece_id=RQOEDOR73GM4W4KNLGPS542JHIIT7XKOZXQ2YVLQR6JCQUPD23IQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.868012671 (59h19m25.299092802s) [noise=true , address=104.36.82.37:4683 , piece_id=CN5FO5OYCPARGFIMP2VSUB2FHJGFYI7WGSEHXAEFTCXIAK3QEW5A, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.900922148 (59h19m25.332002279s) [noise=true , address=104.36.82.2:4734 , piece_id=3GU3VG5HCSTKNLP2L4FHMSWMVQYEVTP3SKAT3WRFCNM7IGZ5XUAA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.935177250 (59h19m25.366257381s) [noise=true , address=104.36.82.35:4586 , piece_id=DKRAQRI6YALBTQ5W5AZPU4D7I6PDLV5NNLHF6NJCZRENBSEYQ5BA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.966459844 (59h19m25.397539975s) [noise=true , address=104.36.82.58:4265 , piece_id=JYMYFO4ZRS7GU3S3YUG2STDZAORUK52TDX74FC2XOETWILKI5NRA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.050105808 (59h19m25.481185939s) [noise=true , address=104.36.82.60:4358 , piece_id=LMZE7L5QDPL7KO6E7S5LTXKVRRZ2MKLJHJ5YYGUXMAFLXXIDRZMQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.059799375 (59h19m25.490879506s) [noise=true , address=104.36.82.62:4427 , piece_id=JA6FYXOYVOW2ABSKW6OSXF52O5ADF34NCELQJMF5MXWY5KURF5OQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.270855508 (59h19m25.701935639s) [noise=true , address=77.90.31.16:4741 , piece_id=ROWNQREQDOXSZ33OCVPUTHCWKKFQ2UJNGA3O7VHLPYNNLLCP7ALQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.363852803 (59h19m25.794932934s) [noise=true , address=104.36.82.23:4456 , piece_id=RXQYJINMERMKG4QRBUP635ELCEM2SEPHXKBN7V62BBCZJLQVRKUA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.386074781 (59h19m25.817154912s) [noise=true , address=104.36.82.38:4721 , piece_id=ABD3F372AYA2CSGX6YIFF4455YEKUA4MCSFH6P4LT3UWQMITWMXQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.425040486 (59h19m25.856120617s) [noise=true , address=45.91.116.11:4136 , piece_id=OXZMVFV6D7I622HOU5DMS7TW5FNJRVCEW4TM7CVCMM62VC6J7FVQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.425075023 (59h19m25.856155154s) [seg_pos={0 0}, upload=61818]: Segment reached optimal threshold of 14 pieces. Aug 11 04:45:31.425249597 (59h19m25.856329758s) [noise=true , address=104.36.82.16:4322 , piece_id=3JFTHIYCW53BZK6UPHAQSALUXC45N3IMPDYVCMA4NJN7C57QIOBQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3): protocol: expected piece hash; context canceled; EOF Aug 11 04:45:31.425348236 (59h19m25.856428367s) [noise=true , address=104.36.82.5:4849 , piece_id=CRQSKVDPYQ2PW52COXWMQWWWWJAMWW2A2SSLNEQWFIASJMYX2PPQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2gSzVmNCEZcfVsFKibjXHs2jMr82D3SWQEykFQQ4uyhhnrj1): protocol: expected piece hash; context canceled; EOF Aug 11 04:45:31.425394103 (59h19m25.856474234s) [noise=true , address=104.36.82.35:4588 , piece_id=SHILYCT7SE26CH26FCFWULB2I5IF5QTBOZXDPGR4YE2X4WFQ5YPQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Aug 11 04:45:31.425471601 (59h19m25.856551742s) [noise=true , address=104.36.82.15:4272 , piece_id=RUWJNFZUWFPPCHKIYKLM65NRZ7URCPY33IWVRI6DNR4L2BLRRRSQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UeKsX3pQfhvCukBvJf1sekK7N6sXkaWaJnZzETKoxyXWo2BFi): protocol: expected piece hash; context canceled; EOF Aug 11 04:45:31.425497581 (59h19m25.856577772s) [seg_pos={0 0}, upload=61818]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 11 04:45:31.425586341 (59h19m25.856666482s) [upload=61818]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 11 04:45:31.425593415 (59h19m25.856673596s) [upload=61818]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:45:31.735520612 (59h19m26.166600743s) [upload=61818]: Done uploading segments... Aug 11 04:45:31.735552864 (59h19m26.166632995s) [upload=61818]: Upload finished. err: <nil> Duration: 9.553902514s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 8.358806911s FAIL (2025-08-11 04:47:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.35882722s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........X............................................X....] duration: 1.849019309s FAIL (2025-08-11 04:47:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 04:44:24.932296741 (59h18m19.363376862s) [upload=61797]: Starting upload Aug 11 04:44:24.932375572 (59h18m19.363455703s) [upload=61797]: Done starting upload Aug 11 04:44:24.932485803 (59h18m19.363565994s) [upload=61797]: Uploading segments... Aug 11 04:44:24.932550086 (59h18m19.363630247s) [upload=61797]: Got next segment. Inline: true Aug 11 04:44:24.932557180 (59h18m19.363637331s) [upload=61797]: Waiting for error group managing segments... Aug 11 04:44:24.932610262 (59h18m19.363690453s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:44:24.932616814 (59h18m19.363696995s) [upload=61797]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:44:24.932621844 (59h18m19.363702005s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:44:25.294615282 (59h18m19.725695413s) [upload=61797]: Done uploading segments... Aug 11 04:44:25.294637325 (59h18m19.725717456s) [upload=61797]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.849046029s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........X............................................X...] duration: 1.824355337s FAIL (2025-08-11 04:46:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 04:44:24.932296741 (59h18m19.363376862s) [upload=61797]: Starting upload Aug 11 04:44:24.932375572 (59h18m19.363455703s) [upload=61797]: Done starting upload Aug 11 04:44:24.932485803 (59h18m19.363565994s) [upload=61797]: Uploading segments... Aug 11 04:44:24.932550086 (59h18m19.363630247s) [upload=61797]: Got next segment. Inline: true Aug 11 04:44:24.932557180 (59h18m19.363637331s) [upload=61797]: Waiting for error group managing segments... Aug 11 04:44:24.932610262 (59h18m19.363690453s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:44:24.932616814 (59h18m19.363696995s) [upload=61797]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:44:24.932621844 (59h18m19.363702005s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:44:25.294615282 (59h18m19.725695413s) [upload=61797]: Done uploading segments... Aug 11 04:44:25.294637325 (59h18m19.725717456s) [upload=61797]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.824385456s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X..............................XX..................X..] duration: 9.620472179s FAIL (2025-08-11 04:46:31 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 11 04:45:27.852954309 (59h19m22.28403444s) [upload=61818]: Starting upload Aug 11 04:45:27.853020255 (59h19m22.284100386s) [upload=61818]: Done starting upload Aug 11 04:45:27.853161035 (59h19m22.284241166s) [upload=61818]: Uploading segments... Aug 11 04:45:27.853265945 (59h19m22.284346066s) [upload=61818]: Got next segment. Inline: false Aug 11 04:45:27.853313025 (59h19m22.284393156s) [upload=61818]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:45:27.853318035 (59h19m22.284398166s) [upload=61818]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:45:28.311646796 (59h19m22.742726927s) [seg_pos={0 0}, upload=61818]: Begin upload segment... Aug 11 04:45:28.311691702 (59h19m22.742771833s) [seg_pos={0 0}, upload=61818]: Done begin upload segment. Aug 11 04:45:28.311716960 (59h19m22.742797081s) [noise=true , address=104.36.82.35:4586 , piece_id=DKRAQRI6YALBTQ5W5AZPU4D7I6PDLV5NNLHF6NJCZRENBSEYQ5BA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.311742379 (59h19m22.74282251s) [noise=true , address=104.36.82.9:4011 , piece_id=BGLEADLDLGTOYQPJV6CLTEGQFQ5L7M5YHFQL674224IKCVFJ4PKA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312118228 (59h19m22.743198349s) [noise=true , address=104.36.82.37:4683 , piece_id=CN5FO5OYCPARGFIMP2VSUB2FHJGFYI7WGSEHXAEFTCXIAK3QEW5A, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312148716 (59h19m22.743228847s) [noise=true , address=104.36.82.23:4456 , piece_id=RXQYJINMERMKG4QRBUP635ELCEM2SEPHXKBN7V62BBCZJLQVRKUA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312386371 (59h19m22.743466503s) [noise=true , address=104.36.82.38:4711 , piece_id=RQOEDOR73GM4W4KNLGPS542JHIIT7XKOZXQ2YVLQR6JCQUPD23IQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312483447 (59h19m22.743563578s) [noise=true , address=104.36.82.15:4272 , piece_id=RUWJNFZUWFPPCHKIYKLM65NRZ7URCPY33IWVRI6DNR4L2BLRRRSQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312662880 (59h19m22.743743032s) [noise=true , address=104.36.82.16:4322 , piece_id=3JFTHIYCW53BZK6UPHAQSALUXC45N3IMPDYVCMA4NJN7C57QIOBQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312840229 (59h19m22.74392036s) [noise=true , address=104.36.82.2:4734 , piece_id=3GU3VG5HCSTKNLP2L4FHMSWMVQYEVTP3SKAT3WRFCNM7IGZ5XUAA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312973755 (59h19m22.744053886s) [noise=true , address=64.49.13.154:4930 , piece_id=CXEIO7OJ75OUXS5RD666FYX5CBP32HMTEJK2TOPFGE7K24Z3HBBQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313147277 (59h19m22.744227398s) [noise=true , address=104.36.82.58:4265 , piece_id=JYMYFO4ZRS7GU3S3YUG2STDZAORUK52TDX74FC2XOETWILKI5NRA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313215117 (59h19m22.744295248s) [noise=true , address=77.90.31.16:4741 , piece_id=ROWNQREQDOXSZ33OCVPUTHCWKKFQ2UJNGA3O7VHLPYNNLLCP7ALQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313409038 (59h19m22.744489169s) [noise=true , address=104.36.82.60:4358 , piece_id=LMZE7L5QDPL7KO6E7S5LTXKVRRZ2MKLJHJ5YYGUXMAFLXXIDRZMQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313454284 (59h19m22.744534415s) [noise=true , address=104.36.82.35:4588 , piece_id=SHILYCT7SE26CH26FCFWULB2I5IF5QTBOZXDPGR4YE2X4WFQ5YPQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313693011 (59h19m22.744773132s) [noise=true , address=104.36.82.62:4427 , piece_id=JA6FYXOYVOW2ABSKW6OSXF52O5ADF34NCELQJMF5MXWY5KURF5OQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313755310 (59h19m22.744835441s) [noise=true , address=104.36.82.5:4849 , piece_id=CRQSKVDPYQ2PW52COXWMQWWWWJAMWW2A2SSLNEQWFIASJMYX2PPQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313910657 (59h19m22.744990789s) [noise=true , address=104.36.82.38:4721 , piece_id=ABD3F372AYA2CSGX6YIFF4455YEKUA4MCSFH6P4LT3UWQMITWMXQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.314072688 (59h19m22.745152819s) [noise=true , address=45.91.116.11:4136 , piece_id=OXZMVFV6D7I622HOU5DMS7TW5FNJRVCEW4TM7CVCMM62VC6J7FVQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.314170785 (59h19m22.745250916s) [noise=true , address=104.36.82.59:4310 , piece_id=EIKGAUOEAG246IR632UUJSEYPYCTQ4JTFR435XAWJVH6ERN5KS2A, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:29.543198877 (59h19m23.974279008s) [upload=61818]: Next returned nil segment Aug 11 04:45:29.543221601 (59h19m23.974301732s) [upload=61818]: Waiting for error group managing segments... Aug 11 04:45:29.949775487 (59h19m24.380855618s) [noise=true , address=64.49.13.154:4930 , piece_id=CXEIO7OJ75OUXS5RD666FYX5CBP32HMTEJK2TOPFGE7K24Z3HBBQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.527562944 (59h19m24.958643075s) [noise=true , address=104.36.82.9:4011 , piece_id=BGLEADLDLGTOYQPJV6CLTEGQFQ5L7M5YHFQL674224IKCVFJ4PKA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.672823547 (59h19m25.103903678s) [noise=true , address=104.36.82.59:4310 , piece_id=EIKGAUOEAG246IR632UUJSEYPYCTQ4JTFR435XAWJVH6ERN5KS2A, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.695652536 (59h19m25.126732667s) [noise=true , address=104.36.82.38:4711 , piece_id=RQOEDOR73GM4W4KNLGPS542JHIIT7XKOZXQ2YVLQR6JCQUPD23IQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.868012671 (59h19m25.299092802s) [noise=true , address=104.36.82.37:4683 , piece_id=CN5FO5OYCPARGFIMP2VSUB2FHJGFYI7WGSEHXAEFTCXIAK3QEW5A, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.900922148 (59h19m25.332002279s) [noise=true , address=104.36.82.2:4734 , piece_id=3GU3VG5HCSTKNLP2L4FHMSWMVQYEVTP3SKAT3WRFCNM7IGZ5XUAA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.935177250 (59h19m25.366257381s) [noise=true , address=104.36.82.35:4586 , piece_id=DKRAQRI6YALBTQ5W5AZPU4D7I6PDLV5NNLHF6NJCZRENBSEYQ5BA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.966459844 (59h19m25.397539975s) [noise=true , address=104.36.82.58:4265 , piece_id=JYMYFO4ZRS7GU3S3YUG2STDZAORUK52TDX74FC2XOETWILKI5NRA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.050105808 (59h19m25.481185939s) [noise=true , address=104.36.82.60:4358 , piece_id=LMZE7L5QDPL7KO6E7S5LTXKVRRZ2MKLJHJ5YYGUXMAFLXXIDRZMQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.059799375 (59h19m25.490879506s) [noise=true , address=104.36.82.62:4427 , piece_id=JA6FYXOYVOW2ABSKW6OSXF52O5ADF34NCELQJMF5MXWY5KURF5OQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.270855508 (59h19m25.701935639s) [noise=true , address=77.90.31.16:4741 , piece_id=ROWNQREQDOXSZ33OCVPUTHCWKKFQ2UJNGA3O7VHLPYNNLLCP7ALQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.363852803 (59h19m25.794932934s) [noise=true , address=104.36.82.23:4456 , piece_id=RXQYJINMERMKG4QRBUP635ELCEM2SEPHXKBN7V62BBCZJLQVRKUA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.386074781 (59h19m25.817154912s) [noise=true , address=104.36.82.38:4721 , piece_id=ABD3F372AYA2CSGX6YIFF4455YEKUA4MCSFH6P4LT3UWQMITWMXQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.425040486 (59h19m25.856120617s) [noise=true , address=45.91.116.11:4136 , piece_id=OXZMVFV6D7I622HOU5DMS7TW5FNJRVCEW4TM7CVCMM62VC6J7FVQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.425075023 (59h19m25.856155154s) [seg_pos={0 0}, upload=61818]: Segment reached optimal threshold of 14 pieces. Aug 11 04:45:31.425249597 (59h19m25.856329758s) [noise=true , address=104.36.82.16:4322 , piece_id=3JFTHIYCW53BZK6UPHAQSALUXC45N3IMPDYVCMA4NJN7C57QIOBQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3): protocol: expected piece hash; context canceled; EOF Aug 11 04:45:31.425348236 (59h19m25.856428367s) [noise=true , address=104.36.82.5:4849 , piece_id=CRQSKVDPYQ2PW52COXWMQWWWWJAMWW2A2SSLNEQWFIASJMYX2PPQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2gSzVmNCEZcfVsFKibjXHs2jMr82D3SWQEykFQQ4uyhhnrj1): protocol: expected piece hash; context canceled; EOF Aug 11 04:45:31.425394103 (59h19m25.856474234s) [noise=true , address=104.36.82.35:4588 , piece_id=SHILYCT7SE26CH26FCFWULB2I5IF5QTBOZXDPGR4YE2X4WFQ5YPQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Aug 11 04:45:31.425471601 (59h19m25.856551742s) [noise=true , address=104.36.82.15:4272 , piece_id=RUWJNFZUWFPPCHKIYKLM65NRZ7URCPY33IWVRI6DNR4L2BLRRRSQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UeKsX3pQfhvCukBvJf1sekK7N6sXkaWaJnZzETKoxyXWo2BFi): protocol: expected piece hash; context canceled; EOF Aug 11 04:45:31.425497581 (59h19m25.856577772s) [seg_pos={0 0}, upload=61818]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 11 04:45:31.425586341 (59h19m25.856666482s) [upload=61818]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 11 04:45:31.425593415 (59h19m25.856673596s) [upload=61818]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:45:31.735520612 (59h19m26.166600743s) [upload=61818]: Done uploading segments... Aug 11 04:45:31.735552864 (59h19m26.166632995s) [upload=61818]: Upload finished. err: <nil> Duration: 9.620501003s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 14.664489335s FAIL (2025-08-11 04:46:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 14.664515827s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X..............................XX..................X.] duration: 13.589742858s FAIL (2025-08-11 04:46:20 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 11 04:45:27.852954309 (59h19m22.28403444s) [upload=61818]: Starting upload Aug 11 04:45:27.853020255 (59h19m22.284100386s) [upload=61818]: Done starting upload Aug 11 04:45:27.853161035 (59h19m22.284241166s) [upload=61818]: Uploading segments... Aug 11 04:45:27.853265945 (59h19m22.284346066s) [upload=61818]: Got next segment. Inline: false Aug 11 04:45:27.853313025 (59h19m22.284393156s) [upload=61818]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:45:27.853318035 (59h19m22.284398166s) [upload=61818]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:45:28.311646796 (59h19m22.742726927s) [seg_pos={0 0}, upload=61818]: Begin upload segment... Aug 11 04:45:28.311691702 (59h19m22.742771833s) [seg_pos={0 0}, upload=61818]: Done begin upload segment. Aug 11 04:45:28.311716960 (59h19m22.742797081s) [noise=true , address=104.36.82.35:4586 , piece_id=DKRAQRI6YALBTQ5W5AZPU4D7I6PDLV5NNLHF6NJCZRENBSEYQ5BA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.311742379 (59h19m22.74282251s) [noise=true , address=104.36.82.9:4011 , piece_id=BGLEADLDLGTOYQPJV6CLTEGQFQ5L7M5YHFQL674224IKCVFJ4PKA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312118228 (59h19m22.743198349s) [noise=true , address=104.36.82.37:4683 , piece_id=CN5FO5OYCPARGFIMP2VSUB2FHJGFYI7WGSEHXAEFTCXIAK3QEW5A, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312148716 (59h19m22.743228847s) [noise=true , address=104.36.82.23:4456 , piece_id=RXQYJINMERMKG4QRBUP635ELCEM2SEPHXKBN7V62BBCZJLQVRKUA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312386371 (59h19m22.743466503s) [noise=true , address=104.36.82.38:4711 , piece_id=RQOEDOR73GM4W4KNLGPS542JHIIT7XKOZXQ2YVLQR6JCQUPD23IQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312483447 (59h19m22.743563578s) [noise=true , address=104.36.82.15:4272 , piece_id=RUWJNFZUWFPPCHKIYKLM65NRZ7URCPY33IWVRI6DNR4L2BLRRRSQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312662880 (59h19m22.743743032s) [noise=true , address=104.36.82.16:4322 , piece_id=3JFTHIYCW53BZK6UPHAQSALUXC45N3IMPDYVCMA4NJN7C57QIOBQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312840229 (59h19m22.74392036s) [noise=true , address=104.36.82.2:4734 , piece_id=3GU3VG5HCSTKNLP2L4FHMSWMVQYEVTP3SKAT3WRFCNM7IGZ5XUAA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312973755 (59h19m22.744053886s) [noise=true , address=64.49.13.154:4930 , piece_id=CXEIO7OJ75OUXS5RD666FYX5CBP32HMTEJK2TOPFGE7K24Z3HBBQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313147277 (59h19m22.744227398s) [noise=true , address=104.36.82.58:4265 , piece_id=JYMYFO4ZRS7GU3S3YUG2STDZAORUK52TDX74FC2XOETWILKI5NRA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313215117 (59h19m22.744295248s) [noise=true , address=77.90.31.16:4741 , piece_id=ROWNQREQDOXSZ33OCVPUTHCWKKFQ2UJNGA3O7VHLPYNNLLCP7ALQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313409038 (59h19m22.744489169s) [noise=true , address=104.36.82.60:4358 , piece_id=LMZE7L5QDPL7KO6E7S5LTXKVRRZ2MKLJHJ5YYGUXMAFLXXIDRZMQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313454284 (59h19m22.744534415s) [noise=true , address=104.36.82.35:4588 , piece_id=SHILYCT7SE26CH26FCFWULB2I5IF5QTBOZXDPGR4YE2X4WFQ5YPQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313693011 (59h19m22.744773132s) [noise=true , address=104.36.82.62:4427 , piece_id=JA6FYXOYVOW2ABSKW6OSXF52O5ADF34NCELQJMF5MXWY5KURF5OQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313755310 (59h19m22.744835441s) [noise=true , address=104.36.82.5:4849 , piece_id=CRQSKVDPYQ2PW52COXWMQWWWWJAMWW2A2SSLNEQWFIASJMYX2PPQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313910657 (59h19m22.744990789s) [noise=true , address=104.36.82.38:4721 , piece_id=ABD3F372AYA2CSGX6YIFF4455YEKUA4MCSFH6P4LT3UWQMITWMXQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.314072688 (59h19m22.745152819s) [noise=true , address=45.91.116.11:4136 , piece_id=OXZMVFV6D7I622HOU5DMS7TW5FNJRVCEW4TM7CVCMM62VC6J7FVQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.314170785 (59h19m22.745250916s) [noise=true , address=104.36.82.59:4310 , piece_id=EIKGAUOEAG246IR632UUJSEYPYCTQ4JTFR435XAWJVH6ERN5KS2A, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:29.543198877 (59h19m23.974279008s) [upload=61818]: Next returned nil segment Aug 11 04:45:29.543221601 (59h19m23.974301732s) [upload=61818]: Waiting for error group managing segments... Aug 11 04:45:29.949775487 (59h19m24.380855618s) [noise=true , address=64.49.13.154:4930 , piece_id=CXEIO7OJ75OUXS5RD666FYX5CBP32HMTEJK2TOPFGE7K24Z3HBBQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.527562944 (59h19m24.958643075s) [noise=true , address=104.36.82.9:4011 , piece_id=BGLEADLDLGTOYQPJV6CLTEGQFQ5L7M5YHFQL674224IKCVFJ4PKA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.672823547 (59h19m25.103903678s) [noise=true , address=104.36.82.59:4310 , piece_id=EIKGAUOEAG246IR632UUJSEYPYCTQ4JTFR435XAWJVH6ERN5KS2A, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.695652536 (59h19m25.126732667s) [noise=true , address=104.36.82.38:4711 , piece_id=RQOEDOR73GM4W4KNLGPS542JHIIT7XKOZXQ2YVLQR6JCQUPD23IQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.868012671 (59h19m25.299092802s) [noise=true , address=104.36.82.37:4683 , piece_id=CN5FO5OYCPARGFIMP2VSUB2FHJGFYI7WGSEHXAEFTCXIAK3QEW5A, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.900922148 (59h19m25.332002279s) [noise=true , address=104.36.82.2:4734 , piece_id=3GU3VG5HCSTKNLP2L4FHMSWMVQYEVTP3SKAT3WRFCNM7IGZ5XUAA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.935177250 (59h19m25.366257381s) [noise=true , address=104.36.82.35:4586 , piece_id=DKRAQRI6YALBTQ5W5AZPU4D7I6PDLV5NNLHF6NJCZRENBSEYQ5BA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.966459844 (59h19m25.397539975s) [noise=true , address=104.36.82.58:4265 , piece_id=JYMYFO4ZRS7GU3S3YUG2STDZAORUK52TDX74FC2XOETWILKI5NRA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.050105808 (59h19m25.481185939s) [noise=true , address=104.36.82.60:4358 , piece_id=LMZE7L5QDPL7KO6E7S5LTXKVRRZ2MKLJHJ5YYGUXMAFLXXIDRZMQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.059799375 (59h19m25.490879506s) [noise=true , address=104.36.82.62:4427 , piece_id=JA6FYXOYVOW2ABSKW6OSXF52O5ADF34NCELQJMF5MXWY5KURF5OQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.270855508 (59h19m25.701935639s) [noise=true , address=77.90.31.16:4741 , piece_id=ROWNQREQDOXSZ33OCVPUTHCWKKFQ2UJNGA3O7VHLPYNNLLCP7ALQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.363852803 (59h19m25.794932934s) [noise=true , address=104.36.82.23:4456 , piece_id=RXQYJINMERMKG4QRBUP635ELCEM2SEPHXKBN7V62BBCZJLQVRKUA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.386074781 (59h19m25.817154912s) [noise=true , address=104.36.82.38:4721 , piece_id=ABD3F372AYA2CSGX6YIFF4455YEKUA4MCSFH6P4LT3UWQMITWMXQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.425040486 (59h19m25.856120617s) [noise=true , address=45.91.116.11:4136 , piece_id=OXZMVFV6D7I622HOU5DMS7TW5FNJRVCEW4TM7CVCMM62VC6J7FVQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.425075023 (59h19m25.856155154s) [seg_pos={0 0}, upload=61818]: Segment reached optimal threshold of 14 pieces. Aug 11 04:45:31.425249597 (59h19m25.856329758s) [noise=true , address=104.36.82.16:4322 , piece_id=3JFTHIYCW53BZK6UPHAQSALUXC45N3IMPDYVCMA4NJN7C57QIOBQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3): protocol: expected piece hash; context canceled; EOF Aug 11 04:45:31.425348236 (59h19m25.856428367s) [noise=true , address=104.36.82.5:4849 , piece_id=CRQSKVDPYQ2PW52COXWMQWWWWJAMWW2A2SSLNEQWFIASJMYX2PPQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2gSzVmNCEZcfVsFKibjXHs2jMr82D3SWQEykFQQ4uyhhnrj1): protocol: expected piece hash; context canceled; EOF Aug 11 04:45:31.425394103 (59h19m25.856474234s) [noise=true , address=104.36.82.35:4588 , piece_id=SHILYCT7SE26CH26FCFWULB2I5IF5QTBOZXDPGR4YE2X4WFQ5YPQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Aug 11 04:45:31.425471601 (59h19m25.856551742s) [noise=true , address=104.36.82.15:4272 , piece_id=RUWJNFZUWFPPCHKIYKLM65NRZ7URCPY33IWVRI6DNR4L2BLRRRSQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UeKsX3pQfhvCukBvJf1sekK7N6sXkaWaJnZzETKoxyXWo2BFi): protocol: expected piece hash; context canceled; EOF Aug 11 04:45:31.425497581 (59h19m25.856577772s) [seg_pos={0 0}, upload=61818]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 11 04:45:31.425586341 (59h19m25.856666482s) [upload=61818]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 11 04:45:31.425593415 (59h19m25.856673596s) [upload=61818]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:45:31.735520612 (59h19m26.166600743s) [upload=61818]: Done uploading segments... Aug 11 04:45:31.735552864 (59h19m26.166632995s) [upload=61818]: Upload finished. err: <nil> Duration: 13.589776121s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 10.181741289s FAIL (2025-08-11 04:45:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 10.181774965s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [............X............................................X..] duration: 1.828607916s FAIL (2025-08-11 04:45:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 04:44:24.932296741 (59h18m19.363376862s) [upload=61797]: Starting upload Aug 11 04:44:24.932375572 (59h18m19.363455703s) [upload=61797]: Done starting upload Aug 11 04:44:24.932485803 (59h18m19.363565994s) [upload=61797]: Uploading segments... Aug 11 04:44:24.932550086 (59h18m19.363630247s) [upload=61797]: Got next segment. Inline: true Aug 11 04:44:24.932557180 (59h18m19.363637331s) [upload=61797]: Waiting for error group managing segments... Aug 11 04:44:24.932610262 (59h18m19.363690453s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:44:24.932616814 (59h18m19.363696995s) [upload=61797]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:44:24.932621844 (59h18m19.363702005s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:44:25.294615282 (59h18m19.725695413s) [upload=61797]: Done uploading segments... Aug 11 04:44:25.294637325 (59h18m19.725717456s) [upload=61797]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.8286316s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X..............................XX..................X] duration: 5.008236737s FAIL (2025-08-11 04:45:32 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 11 04:45:27.852954309 (59h19m22.28403444s) [upload=61818]: Starting upload Aug 11 04:45:27.853020255 (59h19m22.284100386s) [upload=61818]: Done starting upload Aug 11 04:45:27.853161035 (59h19m22.284241166s) [upload=61818]: Uploading segments... Aug 11 04:45:27.853265945 (59h19m22.284346066s) [upload=61818]: Got next segment. Inline: false Aug 11 04:45:27.853313025 (59h19m22.284393156s) [upload=61818]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:45:27.853318035 (59h19m22.284398166s) [upload=61818]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:45:28.311646796 (59h19m22.742726927s) [seg_pos={0 0}, upload=61818]: Begin upload segment... Aug 11 04:45:28.311691702 (59h19m22.742771833s) [seg_pos={0 0}, upload=61818]: Done begin upload segment. Aug 11 04:45:28.311716960 (59h19m22.742797081s) [noise=true , address=104.36.82.35:4586 , piece_id=DKRAQRI6YALBTQ5W5AZPU4D7I6PDLV5NNLHF6NJCZRENBSEYQ5BA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.311742379 (59h19m22.74282251s) [noise=true , address=104.36.82.9:4011 , piece_id=BGLEADLDLGTOYQPJV6CLTEGQFQ5L7M5YHFQL674224IKCVFJ4PKA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312118228 (59h19m22.743198349s) [noise=true , address=104.36.82.37:4683 , piece_id=CN5FO5OYCPARGFIMP2VSUB2FHJGFYI7WGSEHXAEFTCXIAK3QEW5A, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312148716 (59h19m22.743228847s) [noise=true , address=104.36.82.23:4456 , piece_id=RXQYJINMERMKG4QRBUP635ELCEM2SEPHXKBN7V62BBCZJLQVRKUA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312386371 (59h19m22.743466503s) [noise=true , address=104.36.82.38:4711 , piece_id=RQOEDOR73GM4W4KNLGPS542JHIIT7XKOZXQ2YVLQR6JCQUPD23IQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312483447 (59h19m22.743563578s) [noise=true , address=104.36.82.15:4272 , piece_id=RUWJNFZUWFPPCHKIYKLM65NRZ7URCPY33IWVRI6DNR4L2BLRRRSQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312662880 (59h19m22.743743032s) [noise=true , address=104.36.82.16:4322 , piece_id=3JFTHIYCW53BZK6UPHAQSALUXC45N3IMPDYVCMA4NJN7C57QIOBQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312840229 (59h19m22.74392036s) [noise=true , address=104.36.82.2:4734 , piece_id=3GU3VG5HCSTKNLP2L4FHMSWMVQYEVTP3SKAT3WRFCNM7IGZ5XUAA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.312973755 (59h19m22.744053886s) [noise=true , address=64.49.13.154:4930 , piece_id=CXEIO7OJ75OUXS5RD666FYX5CBP32HMTEJK2TOPFGE7K24Z3HBBQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313147277 (59h19m22.744227398s) [noise=true , address=104.36.82.58:4265 , piece_id=JYMYFO4ZRS7GU3S3YUG2STDZAORUK52TDX74FC2XOETWILKI5NRA, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313215117 (59h19m22.744295248s) [noise=true , address=77.90.31.16:4741 , piece_id=ROWNQREQDOXSZ33OCVPUTHCWKKFQ2UJNGA3O7VHLPYNNLLCP7ALQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313409038 (59h19m22.744489169s) [noise=true , address=104.36.82.60:4358 , piece_id=LMZE7L5QDPL7KO6E7S5LTXKVRRZ2MKLJHJ5YYGUXMAFLXXIDRZMQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313454284 (59h19m22.744534415s) [noise=true , address=104.36.82.35:4588 , piece_id=SHILYCT7SE26CH26FCFWULB2I5IF5QTBOZXDPGR4YE2X4WFQ5YPQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313693011 (59h19m22.744773132s) [noise=true , address=104.36.82.62:4427 , piece_id=JA6FYXOYVOW2ABSKW6OSXF52O5ADF34NCELQJMF5MXWY5KURF5OQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313755310 (59h19m22.744835441s) [noise=true , address=104.36.82.5:4849 , piece_id=CRQSKVDPYQ2PW52COXWMQWWWWJAMWW2A2SSLNEQWFIASJMYX2PPQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.313910657 (59h19m22.744990789s) [noise=true , address=104.36.82.38:4721 , piece_id=ABD3F372AYA2CSGX6YIFF4455YEKUA4MCSFH6P4LT3UWQMITWMXQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.314072688 (59h19m22.745152819s) [noise=true , address=45.91.116.11:4136 , piece_id=OXZMVFV6D7I622HOU5DMS7TW5FNJRVCEW4TM7CVCMM62VC6J7FVQ, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:28.314170785 (59h19m22.745250916s) [noise=true , address=104.36.82.59:4310 , piece_id=EIKGAUOEAG246IR632UUJSEYPYCTQ4JTFR435XAWJVH6ERN5KS2A, seg_pos={0 0}, upload=61818]: Uploading piece... Aug 11 04:45:29.543198877 (59h19m23.974279008s) [upload=61818]: Next returned nil segment Aug 11 04:45:29.543221601 (59h19m23.974301732s) [upload=61818]: Waiting for error group managing segments... Aug 11 04:45:29.949775487 (59h19m24.380855618s) [noise=true , address=64.49.13.154:4930 , piece_id=CXEIO7OJ75OUXS5RD666FYX5CBP32HMTEJK2TOPFGE7K24Z3HBBQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.527562944 (59h19m24.958643075s) [noise=true , address=104.36.82.9:4011 , piece_id=BGLEADLDLGTOYQPJV6CLTEGQFQ5L7M5YHFQL674224IKCVFJ4PKA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.672823547 (59h19m25.103903678s) [noise=true , address=104.36.82.59:4310 , piece_id=EIKGAUOEAG246IR632UUJSEYPYCTQ4JTFR435XAWJVH6ERN5KS2A, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.695652536 (59h19m25.126732667s) [noise=true , address=104.36.82.38:4711 , piece_id=RQOEDOR73GM4W4KNLGPS542JHIIT7XKOZXQ2YVLQR6JCQUPD23IQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.868012671 (59h19m25.299092802s) [noise=true , address=104.36.82.37:4683 , piece_id=CN5FO5OYCPARGFIMP2VSUB2FHJGFYI7WGSEHXAEFTCXIAK3QEW5A, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.900922148 (59h19m25.332002279s) [noise=true , address=104.36.82.2:4734 , piece_id=3GU3VG5HCSTKNLP2L4FHMSWMVQYEVTP3SKAT3WRFCNM7IGZ5XUAA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.935177250 (59h19m25.366257381s) [noise=true , address=104.36.82.35:4586 , piece_id=DKRAQRI6YALBTQ5W5AZPU4D7I6PDLV5NNLHF6NJCZRENBSEYQ5BA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:30.966459844 (59h19m25.397539975s) [noise=true , address=104.36.82.58:4265 , piece_id=JYMYFO4ZRS7GU3S3YUG2STDZAORUK52TDX74FC2XOETWILKI5NRA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.050105808 (59h19m25.481185939s) [noise=true , address=104.36.82.60:4358 , piece_id=LMZE7L5QDPL7KO6E7S5LTXKVRRZ2MKLJHJ5YYGUXMAFLXXIDRZMQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.059799375 (59h19m25.490879506s) [noise=true , address=104.36.82.62:4427 , piece_id=JA6FYXOYVOW2ABSKW6OSXF52O5ADF34NCELQJMF5MXWY5KURF5OQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.270855508 (59h19m25.701935639s) [noise=true , address=77.90.31.16:4741 , piece_id=ROWNQREQDOXSZ33OCVPUTHCWKKFQ2UJNGA3O7VHLPYNNLLCP7ALQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.363852803 (59h19m25.794932934s) [noise=true , address=104.36.82.23:4456 , piece_id=RXQYJINMERMKG4QRBUP635ELCEM2SEPHXKBN7V62BBCZJLQVRKUA, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.386074781 (59h19m25.817154912s) [noise=true , address=104.36.82.38:4721 , piece_id=ABD3F372AYA2CSGX6YIFF4455YEKUA4MCSFH6P4LT3UWQMITWMXQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.425040486 (59h19m25.856120617s) [noise=true , address=45.91.116.11:4136 , piece_id=OXZMVFV6D7I622HOU5DMS7TW5FNJRVCEW4TM7CVCMM62VC6J7FVQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: <nil> Aug 11 04:45:31.425075023 (59h19m25.856155154s) [seg_pos={0 0}, upload=61818]: Segment reached optimal threshold of 14 pieces. Aug 11 04:45:31.425249597 (59h19m25.856329758s) [noise=true , address=104.36.82.16:4322 , piece_id=3JFTHIYCW53BZK6UPHAQSALUXC45N3IMPDYVCMA4NJN7C57QIOBQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WwYqhEYC9QPkGDiSt5bF7izvhpsawWx5dwTqP5zZcPyigs1n3): protocol: expected piece hash; context canceled; EOF Aug 11 04:45:31.425348236 (59h19m25.856428367s) [noise=true , address=104.36.82.5:4849 , piece_id=CRQSKVDPYQ2PW52COXWMQWWWWJAMWW2A2SSLNEQWFIASJMYX2PPQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2gSzVmNCEZcfVsFKibjXHs2jMr82D3SWQEykFQQ4uyhhnrj1): protocol: expected piece hash; context canceled; EOF Aug 11 04:45:31.425394103 (59h19m25.856474234s) [noise=true , address=104.36.82.35:4588 , piece_id=SHILYCT7SE26CH26FCFWULB2I5IF5QTBOZXDPGR4YE2X4WFQ5YPQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Aug 11 04:45:31.425471601 (59h19m25.856551742s) [noise=true , address=104.36.82.15:4272 , piece_id=RUWJNFZUWFPPCHKIYKLM65NRZ7URCPY33IWVRI6DNR4L2BLRRRSQ, seg_pos={0 0}, upload=61818]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UeKsX3pQfhvCukBvJf1sekK7N6sXkaWaJnZzETKoxyXWo2BFi): protocol: expected piece hash; context canceled; EOF Aug 11 04:45:31.425497581 (59h19m25.856577772s) [seg_pos={0 0}, upload=61818]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 11 04:45:31.425586341 (59h19m25.856666482s) [upload=61818]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 11 04:45:31.425593415 (59h19m25.856673596s) [upload=61818]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:45:31.735520612 (59h19m26.166600743s) [upload=61818]: Done uploading segments... Aug 11 04:45:31.735552864 (59h19m26.166632995s) [upload=61818]: Upload finished. err: <nil> Duration: 5.008261345s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X..............................XX..................] duration: 8.011245452s FAIL (2025-08-11 04:45:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 04:31:02.553200391 (59h4m56.984280532s) [upload=61567]: Starting upload Aug 11 04:31:02.553250596 (59h4m56.984330727s) [upload=61567]: Done starting upload Aug 11 04:31:02.553416453 (59h4m56.984496584s) [upload=61567]: Uploading segments... Aug 11 04:31:02.554057240 (59h4m56.985137371s) [upload=61567]: Got next segment. Inline: false Aug 11 04:31:02.554108729 (59h4m56.98518886s) [upload=61567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:31:02.554113778 (59h4m56.985193909s) [upload=61567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:31:02.769342558 (59h4m57.200422689s) [upload=61567]: Next segment err: metaclient: Too Many Requests Aug 11 04:31:02.769369820 (59h4m57.200449951s) [upload=61567]: Done uploading segments... Aug 11 04:31:02.769408495 (59h4m57.200488626s) [upload=61567]: Upload finished. err: metaclient: Too Many Requests Duration: 8.011273545s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 7.495630874s FAIL (2025-08-11 04:45:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 11 04:14:05.318440227 (58h47m59.749520358s) [upload=61279]: Starting upload Aug 11 04:14:05.318493429 (58h47m59.74957356s) [upload=61279]: Done starting upload Aug 11 04:14:05.318549967 (58h47m59.749630098s) [upload=61279]: Uploading segments... Aug 11 04:14:05.318671200 (58h47m59.749751331s) [upload=61279]: Got next segment. Inline: false Aug 11 04:14:05.318697891 (58h47m59.749778022s) [upload=61279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:14:05.318702129 (58h47m59.74978226s) [upload=61279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:14:05.765239352 (58h48m0.196319493s) [seg_pos={0 0}, upload=61279]: Begin upload segment... Aug 11 04:14:05.765296240 (58h48m0.196376371s) [seg_pos={0 0}, upload=61279]: Done begin upload segment. Aug 11 04:14:05.765312542 (58h48m0.196392673s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765316890 (58h48m0.196397021s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765670567 (58h48m0.196750688s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.765995629 (58h48m0.19707576s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766263110 (58h48m0.197343241s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766547765 (58h48m0.197627896s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.766877987 (58h48m0.197958118s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767100553 (58h48m0.198180684s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767302078 (58h48m0.198382209s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767522400 (58h48m0.198602521s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767725739 (58h48m0.19880587s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.767932685 (58h48m0.199012816s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768173816 (58h48m0.199253957s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768438583 (58h48m0.199518714s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768667711 (58h48m0.199747842s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:05.768927238 (58h48m0.200007369s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Uploading piece... Aug 11 04:14:08.309487348 (58h48m2.740567479s) [upload=61279]: Next returned nil segment Aug 11 04:14:08.309537213 (58h48m2.740617344s) [upload=61279]: Waiting for error group managing segments... Aug 11 04:14:09.205648620 (58h48m3.636728751s) [noise=true , address=104.36.82.10:4079 , piece_id=POPE63FDLNL7H6W6S4LAUOSEGCUV4ITOET6SV6A4YJBOLX7B5GZQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.280236001 (58h48m3.711316122s) [noise=true , address=77.90.31.14:4663 , piece_id=XI2Q7T2ARKZIB3X5JAKGXDMVUKZ565MMEHFRTNZWLWPZC76FQPLQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.426409640 (58h48m3.857489761s) [noise=true , address=104.36.82.13:4186 , piece_id=TDPQ6AKV5DR5JVDHEXCI72IGYEYVDXB3PYR7WOXUTAOXMLQYMDDA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:09.536862883 (58h48m3.967943004s) [noise=true , address=45.91.116.11:4119 , piece_id=GY6REQIUQ7KUKM7622B2N53FYX3OGR3WWUO5LH35MA6XDOPWBCYQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:11.592695751 (58h48m6.023775883s) [noise=true , address=104.36.82.3:4767 , piece_id=Q6YEXCB3CQZYXQ54ITILSFU5LFCV5S6PVVFTMSHJ4UJG6DCMZZCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:13.145726321 (58h48m7.576806452s) [noise=true , address=45.91.116.10:4081 , piece_id=JB2IBDZ7FMTMLUZ23JSCJWHXSUDS2ZPCMZHSY5WFCQROR7OGACOQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:14.206618488 (58h48m8.637698619s) [noise=true , address=45.91.116.13:4154 , piece_id=ZQL7ZRSQKV7U2VNKBCFPH5AXWQIMG445AXTAPZEPRTB7GPQKG7DA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.326267188 (58h48m12.757347309s) [noise=true , address=77.90.31.11:4554 , piece_id=QTGF7Q4CFDYENXAFHSPW4QFCCS6YC2FF3AUFGY4KBXNJYS7JF7TQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:18.346485323 (58h48m12.777565454s) [noise=true , address=45.91.116.4:4938 , piece_id=OV27IPAB35KRP5CAASYQEHFCHGQ2Z6DSKAGFRIKV237BWKCQBGBQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:19.027231925 (58h48m13.458312056s) [noise=true , address=77.90.31.12:4592 , piece_id=VNK25A4A3N452RJCX5R67NCNMA6VXESSV3YIBBUCGBJPONXYLCRQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:21.696292999 (58h48m16.12737313s) [noise=true , address=45.91.116.7:4010 , piece_id=LBWCSEOSBBSJGHZWKH4D3TCBFQ67NPH5MWIVUC6CT3JNMAFLBEVQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:27.199043519 (58h48m21.63012364s) [noise=true , address=45.91.116.5:4297 , piece_id=C3NVLER4JE4TTUNCC4YZ5CJXS7G7DJG5PBHWGCEOW6GRVVDX555Q, seg_pos={0 0}, upload=61279]: Done uploading piece. err: <nil> Aug 11 04:14:34.675838024 (58h48m29.106918155s) [noise=true , address=77.90.31.16:4733 , piece_id=5PU6HVEZGTAWHFF2HK7E2ECCONEQGG6BJL5VTP7AM4C7DF2GGGCQ, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675910663 (58h48m29.106990794s) [noise=true , address=104.36.82.7:4910 , piece_id=SDOK67NTIXR6HIZZVPBMAEVD4XO2ZGK7S5VJFSLRAP6JE5VZQIKA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.675978763 (58h48m29.107058894s) [noise=true , address=104.36.82.8:4970 , piece_id=LWGBQF6BHVYU6V5BSYOXPQ6SDY22OFV5GTEDW6OCL5CCQPUB2WUA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: expected piece hash; context deadline exceeded; EOF Aug 11 04:14:34.676097411 (58h48m29.107177542s) [noise=true , address=77.90.31.19:4842 , piece_id=VVUBRZE4A3JKRYXW5DWYTWK6XXFYTGLW2AXCIKJWJDF4QZVUXGZA, seg_pos={0 0}, upload=61279]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47230->77.90.31.19:4842: use of closed network connection Aug 11 04:14:34.676133039 (58h48m29.10721317s) [seg_pos={0 0}, upload=61279]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 11 04:14:34.904930895 (58h48m29.336011016s) [upload=61279]: Done uploading segments... Aug 11 04:14:34.904958387 (58h48m29.336038518s) [upload=61279]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.495659289s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............X............................................X.] duration: 1.820896862s FAIL (2025-08-11 04:44:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 04:44:24.932296741 (59h18m19.363376862s) [upload=61797]: Starting upload Aug 11 04:44:24.932375572 (59h18m19.363455703s) [upload=61797]: Done starting upload Aug 11 04:44:24.932485803 (59h18m19.363565994s) [upload=61797]: Uploading segments... Aug 11 04:44:24.932550086 (59h18m19.363630247s) [upload=61797]: Got next segment. Inline: true Aug 11 04:44:24.932557180 (59h18m19.363637331s) [upload=61797]: Waiting for error group managing segments... Aug 11 04:44:24.932610262 (59h18m19.363690453s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:44:24.932616814 (59h18m19.363696995s) [upload=61797]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:44:24.932621844 (59h18m19.363702005s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:44:25.294615282 (59h18m19.725695413s) [upload=61797]: Done uploading segments... Aug 11 04:44:25.294637325 (59h18m19.725717456s) [upload=61797]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.820926308s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X..............................XX.................] duration: 8.864307781s FAIL (2025-08-11 04:44:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 04:31:02.553200391 (59h4m56.984280532s) [upload=61567]: Starting upload Aug 11 04:31:02.553250596 (59h4m56.984330727s) [upload=61567]: Done starting upload Aug 11 04:31:02.553416453 (59h4m56.984496584s) [upload=61567]: Uploading segments... Aug 11 04:31:02.554057240 (59h4m56.985137371s) [upload=61567]: Got next segment. Inline: false Aug 11 04:31:02.554108729 (59h4m56.98518886s) [upload=61567]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:31:02.554113778 (59h4m56.985193909s) [upload=61567]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 11 04:31:02.769342558 (59h4m57.200422689s) [upload=61567]: Next segment err: metaclient: Too Many Requests Aug 11 04:31:02.769369820 (59h4m57.200449951s) [upload=61567]: Done uploading segments... Aug 11 04:31:02.769408495 (59h4m57.200488626s) [upload=61567]: Upload finished. err: metaclient: Too Many Requests Duration: 8.864347557s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............X............................................X] duration: 1.117524831s FAIL (2025-08-11 04:44:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 11 04:44:24.932296741 (59h18m19.363376862s) [upload=61797]: Starting upload Aug 11 04:44:24.932375572 (59h18m19.363455703s) [upload=61797]: Done starting upload Aug 11 04:44:24.932485803 (59h18m19.363565994s) [upload=61797]: Uploading segments... Aug 11 04:44:24.932550086 (59h18m19.363630247s) [upload=61797]: Got next segment. Inline: true Aug 11 04:44:24.932557180 (59h18m19.363637331s) [upload=61797]: Waiting for error group managing segments... Aug 11 04:44:24.932610262 (59h18m19.363690453s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 11 04:44:24.932616814 (59h18m19.363696995s) [upload=61797]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 11 04:44:24.932621844 (59h18m19.363702005s) [upload=61797]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 11 04:44:25.294615282 (59h18m19.725695413s) [upload=61797]: Done uploading segments... Aug 11 04:44:25.294637325 (59h18m19.725717456s) [upload=61797]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.117549488s core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)