Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.217128774s FAIL (2025-07-04 23:03:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.217146638s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.796154929s FAIL (2025-07-04 23:03:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.796178874s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.289624014s FAIL (2025-07-04 23:03:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.289641156s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.799495925s FAIL (2025-07-04 23:03:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.799525884s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.26747921s FAIL (2025-07-04 23:02:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.267502917s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.842619698s FAIL (2025-07-04 23:02:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.842648893s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.310694503s FAIL (2025-07-04 23:01:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.310718369s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.725923755s FAIL (2025-07-04 23:01:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.725949984s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.380211575s FAIL (2025-07-04 23:00:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.3808883s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.368039542s FAIL (2025-07-04 23:00:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.368060491s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.606938976s FAIL (2025-07-04 22:59:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.606970007s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.27712054s FAIL (2025-07-04 22:59:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.277142304s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.852614869s FAIL (2025-07-04 22:58:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.852644606s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.229438032s FAIL (2025-07-04 22:58:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.229497866s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.852289353s FAIL (2025-07-04 22:58:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.852310793s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.283002998s FAIL (2025-07-04 22:57:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.283031072s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.678773196s FAIL (2025-07-04 22:57:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.678808264s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.691611168s FAIL (2025-07-04 22:57:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.691636545s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.297072464s FAIL (2025-07-04 22:56:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.297095899s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.247963234s FAIL (2025-07-04 22:56:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.247982732s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.273766627s FAIL (2025-07-04 22:55:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.273785303s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.287679181s FAIL (2025-07-04 22:54:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.287900445s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.393150316s FAIL (2025-07-04 22:54:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.393172709s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.251029037s FAIL (2025-07-04 22:53:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.251055347s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.245177485s FAIL (2025-07-04 22:53:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.245199188s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000638788s FAIL (2025-07-04 22:52:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000659719s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.24983124s FAIL (2025-07-04 22:25:19 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51362->45.38.24.2:443: read: connection reset by peer Duration: 2.249854796s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.270273415s FAIL (2025-07-04 22:24:36 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51362->45.38.24.2:443: read: connection reset by peer Duration: 2.270300839s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.220550143s FAIL (2025-07-04 22:23:25 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51362->45.38.24.2:443: read: connection reset by peer Duration: 2.220578266s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.23356489s FAIL (2025-07-04 22:22:57 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51362->45.38.24.2:443: read: connection reset by peer Duration: 2.233579718s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.28631745s FAIL (2025-07-04 22:22:32 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51362->45.38.24.2:443: read: connection reset by peer Duration: 2.286333681s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.359017204s FAIL (2025-07-04 22:21:33 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51362->45.38.24.2:443: read: connection reset by peer Duration: 2.359041833s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.257467548s FAIL (2025-07-04 22:20:23 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51362->45.38.24.2:443: read: connection reset by peer Duration: 2.257488099s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.309741807s FAIL (2025-07-04 22:19:41 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51362->45.38.24.2:443: read: connection reset by peer Duration: 2.309762306s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.278319705s FAIL (2025-07-04 22:18:54 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51362->45.38.24.2:443: read: connection reset by peer Duration: 2.278340293s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.307641406s FAIL (2025-07-04 22:18:11 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51362->45.38.24.2:443: read: connection reset by peer Duration: 2.30766408s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.215499161s FAIL (2025-07-04 22:17:15 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51362->45.38.24.2:443: read: connection reset by peer Duration: 2.215516584s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.318275483s FAIL (2025-07-04 22:16:06 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51362->45.38.24.2:443: read: connection reset by peer Duration: 2.318298205s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.309160664s FAIL (2025-07-04 22:15:12 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51362->45.38.24.2:443: read: connection reset by peer Duration: 2.309189899s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.299949068s FAIL (2025-07-04 22:14:09 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51362->45.38.24.2:443: read: connection reset by peer Duration: 2.299973624s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.241663963s FAIL (2025-07-04 22:13:11 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51362->45.38.24.2:443: read: connection reset by peer Duration: 2.241681557s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.285101801s FAIL (2025-07-04 22:12:04 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51362->45.38.24.2:443: read: connection reset by peer Duration: 2.285121829s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.263401629s FAIL (2025-07-04 22:11:24 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51362->45.38.24.2:443: read: connection reset by peer Duration: 2.263425775s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.244670734s FAIL (2025-07-04 22:10:10 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51362->45.38.24.2:443: read: connection reset by peer Duration: 2.244692588s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.25518421s FAIL (2025-07-04 22:09:21 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51362->45.38.24.2:443: read: connection reset by peer Duration: 2.255207173s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.221166388s FAIL (2025-07-04 22:08:55 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51362->45.38.24.2:443: read: connection reset by peer Duration: 2.221187799s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.314306536s FAIL (2025-07-04 22:08:39 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51362->45.38.24.2:443: read: connection reset by peer Duration: 2.314333999s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.294278369s FAIL (2025-07-04 22:07:33 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51362->45.38.24.2:443: read: connection reset by peer Duration: 2.294298497s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.339217894s FAIL (2025-07-04 22:06:30 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51362->45.38.24.2:443: read: connection reset by peer Duration: 2.339234937s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.289172051s FAIL (2025-07-04 22:05:59 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:51362->45.38.24.2:443: read: connection reset by peer Duration: 2.289195796s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)