Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.657311577s FAIL (2025-07-04 23:11:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.657338489s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.223698507s FAIL (2025-07-04 23:11:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.22371583s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.731448246s FAIL (2025-07-04 23:10:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.731473964s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.26768234s FAIL (2025-07-04 23:10:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.267702648s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.994361812s FAIL (2025-07-04 23:10:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.994409243s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.06386059s FAIL (2025-07-04 23:09:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.063884145s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.267882041s FAIL (2025-07-04 23:09:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.267899765s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.257520102s FAIL (2025-07-04 23:09:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.257542965s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.662102752s FAIL (2025-07-04 23:08:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.662132508s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.913533621s FAIL (2025-07-04 23:08:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.913560802s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.265729783s FAIL (2025-07-04 23:08:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.266031481s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.237493666s FAIL (2025-07-04 23:07:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.237520908s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.682126344s FAIL (2025-07-04 23:07:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.682157773s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.388384183s FAIL (2025-07-04 23:07:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.388427075s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.768031424s FAIL (2025-07-04 23:06:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.768052183s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.324254096s FAIL (2025-07-04 23:06:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.324274995s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.25378073s FAIL (2025-07-04 23:06:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.253794186s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.842262603s FAIL (2025-07-04 23:06:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.842285578s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.272641194s FAIL (2025-07-04 23:05:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.27266526s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.327919629s FAIL (2025-07-04 23:05:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.327938685s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.788654376s FAIL (2025-07-04 23:05:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.788682531s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.293615569s FAIL (2025-07-04 23:04:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.293630037s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.017908017s FAIL (2025-07-04 23:04:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.017940157s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.271407444s FAIL (2025-07-04 23:04:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.271429857s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.217128774s FAIL (2025-07-04 23:03:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.217146638s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.796154929s FAIL (2025-07-04 23:03:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.796178874s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.289624014s FAIL (2025-07-04 23:03:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.289641156s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.799495925s FAIL (2025-07-04 23:03:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.799525884s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.26747921s FAIL (2025-07-04 23:02:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.267502917s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.842619698s FAIL (2025-07-04 23:02:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.842648893s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.310694503s FAIL (2025-07-04 23:01:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.310718369s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.725923755s FAIL (2025-07-04 23:01:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.725949984s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.380211575s FAIL (2025-07-04 23:00:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.3808883s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.368039542s FAIL (2025-07-04 23:00:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.368060491s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.606938976s FAIL (2025-07-04 22:59:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.606970007s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.27712054s FAIL (2025-07-04 22:59:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.277142304s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.852614869s FAIL (2025-07-04 22:58:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.852644606s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.229438032s FAIL (2025-07-04 22:58:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.229497866s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.852289353s FAIL (2025-07-04 22:58:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.852310793s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.283002998s FAIL (2025-07-04 22:57:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.283031072s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.678773196s FAIL (2025-07-04 22:57:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.678808264s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.691611168s FAIL (2025-07-04 22:57:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.691636545s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.297072464s FAIL (2025-07-04 22:56:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.297095899s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.247963234s FAIL (2025-07-04 22:56:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.247982732s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.273766627s FAIL (2025-07-04 22:55:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.273785303s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.287679181s FAIL (2025-07-04 22:54:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.287900445s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.393150316s FAIL (2025-07-04 22:54:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.393172709s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.251029037s FAIL (2025-07-04 22:53:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.251055347s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.245177485s FAIL (2025-07-04 22:53:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.245199188s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000638788s FAIL (2025-07-04 22:52:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000659719s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)