Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.883972541s FAIL (2025-07-04 23:37:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.883991918s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.68949822s FAIL (2025-07-04 23:36:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.689526834s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.726116983s FAIL (2025-07-04 23:35:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.726146839s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.615570662s FAIL (2025-07-04 23:34:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.615608245s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.133849156s FAIL (2025-07-04 23:33:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.133872329s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.676428394s FAIL (2025-07-04 23:33:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.676448704s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.252895209s FAIL (2025-07-04 23:33:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.252912992s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.245037618s FAIL (2025-07-04 23:32:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.245060012s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.700545583s FAIL (2025-07-04 23:32:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.700573286s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.303578259s FAIL (2025-07-04 23:32:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.30359965s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.205701152s FAIL (2025-07-04 23:32:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.205730227s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.643353974s FAIL (2025-07-04 23:32:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.643378781s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.988567217s FAIL (2025-07-04 23:31:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.988596382s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.249865328s FAIL (2025-07-04 23:30:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.249888582s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.799151242s FAIL (2025-07-04 23:30:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.799180328s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.301100769s FAIL (2025-07-04 23:30:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.301121208s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.093684493s FAIL (2025-07-04 23:29:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.09370884s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.22116563s FAIL (2025-07-04 23:29:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.22118734s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.284938129s FAIL (2025-07-04 23:28:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.284965903s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.665151733s FAIL (2025-07-04 23:28:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.665182382s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.249960155s FAIL (2025-07-04 23:28:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.249986536s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 9.095445825s FAIL (2025-07-04 23:27:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 9.09546877s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.325187831s FAIL (2025-07-04 23:27:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.325209142s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.734779164s FAIL (2025-07-04 23:26:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.73480925s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.745549458s FAIL (2025-07-04 23:26:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.745572002s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.311888147s FAIL (2025-07-04 23:26:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.311906803s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.486266937s FAIL (2025-07-04 23:25:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.486286574s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.748410861s FAIL (2025-07-04 23:25:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.748433213s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.349771578s FAIL (2025-07-04 23:25:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.349793348s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.865391175s FAIL (2025-07-04 23:24:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.865420812s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.603014219s FAIL (2025-07-04 23:24:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.603046711s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.243142372s FAIL (2025-07-04 23:24:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.24316254s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.704393987s FAIL (2025-07-04 23:23:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.704436047s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.412221344s FAIL (2025-07-04 23:23:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.412245359s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.734063694s FAIL (2025-07-04 23:22:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.7340925s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.919319835s FAIL (2025-07-04 23:22:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.919346545s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.233136984s FAIL (2025-07-04 23:21:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.233159667s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.244592006s FAIL (2025-07-04 23:21:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.244612716s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.785184741s FAIL (2025-07-04 23:20:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.785213397s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.259264458s FAIL (2025-07-04 23:20:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.259309244s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.707738048s FAIL (2025-07-04 23:20:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.707763537s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.714502106s FAIL (2025-07-04 23:19:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.714532384s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.320376537s FAIL (2025-07-04 23:19:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.320390904s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.92633037s FAIL (2025-07-04 23:19:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.926363803s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.311654663s FAIL (2025-07-04 23:18:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.311670443s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.67142499s FAIL (2025-07-04 23:18:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.671449098s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.951944604s FAIL (2025-07-04 23:17:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.951960184s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.265394063s FAIL (2025-07-04 23:17:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.265414072s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 13.646095435s FAIL (2025-07-04 23:17:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 13.646122958s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.253574339s FAIL (2025-07-04 23:17:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.253592215s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)