Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.233136984s FAIL (2025-07-04 23:21:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.233159667s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.244592006s FAIL (2025-07-04 23:21:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.244612716s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.785184741s FAIL (2025-07-04 23:20:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.785213397s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.259264458s FAIL (2025-07-04 23:20:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.259309244s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.707738048s FAIL (2025-07-04 23:20:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.707763537s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.714502106s FAIL (2025-07-04 23:19:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.714532384s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.320376537s FAIL (2025-07-04 23:19:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.320390904s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.92633037s FAIL (2025-07-04 23:19:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.926363803s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.311654663s FAIL (2025-07-04 23:18:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.311670443s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.67142499s FAIL (2025-07-04 23:18:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.671449098s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.951944604s FAIL (2025-07-04 23:17:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.951960184s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.265394063s FAIL (2025-07-04 23:17:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.265414072s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 13.646095435s FAIL (2025-07-04 23:17:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 13.646122958s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.253574339s FAIL (2025-07-04 23:17:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.253592215s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.993182483s FAIL (2025-07-04 23:16:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.993231697s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.285701139s FAIL (2025-07-04 23:16:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.285725836s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.352295291s FAIL (2025-07-04 23:15:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.352317003s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.19120319s FAIL (2025-07-04 23:15:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.191231785s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.449228543s FAIL (2025-07-04 23:14:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.449251226s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.29161821s FAIL (2025-07-04 23:14:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.291646223s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.856327481s FAIL (2025-07-04 23:14:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.856356577s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.291713864s FAIL (2025-07-04 23:13:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.291743599s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.393882s FAIL (2025-07-04 23:13:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.393905154s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.586334343s FAIL (2025-07-04 23:13:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.586359752s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.27657432s FAIL (2025-07-04 23:12:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.276601853s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.240899782s FAIL (2025-07-04 23:12:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.240918147s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.657311577s FAIL (2025-07-04 23:11:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.657338489s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.223698507s FAIL (2025-07-04 23:11:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.22371583s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.731448246s FAIL (2025-07-04 23:10:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.731473964s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.26768234s FAIL (2025-07-04 23:10:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.267702648s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.994361812s FAIL (2025-07-04 23:10:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.994409243s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.06386059s FAIL (2025-07-04 23:09:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.063884145s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.267882041s FAIL (2025-07-04 23:09:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.267899765s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.257520102s FAIL (2025-07-04 23:09:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.257542965s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.662102752s FAIL (2025-07-04 23:08:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.662132508s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.913533621s FAIL (2025-07-04 23:08:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.913560802s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.265729783s FAIL (2025-07-04 23:08:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.266031481s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.237493666s FAIL (2025-07-04 23:07:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.237520908s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.682126344s FAIL (2025-07-04 23:07:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.682157773s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.388384183s FAIL (2025-07-04 23:07:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.388427075s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.768031424s FAIL (2025-07-04 23:06:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.768052183s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.324254096s FAIL (2025-07-04 23:06:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.324274995s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.25378073s FAIL (2025-07-04 23:06:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.253794186s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.842262603s FAIL (2025-07-04 23:06:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.842285578s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.272641194s FAIL (2025-07-04 23:05:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.27266526s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.327919629s FAIL (2025-07-04 23:05:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.327938685s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.788654376s FAIL (2025-07-04 23:05:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.788682531s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.293615569s FAIL (2025-07-04 23:04:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.293630037s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.017908017s FAIL (2025-07-04 23:04:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== pload=222113]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 4 22:55:22.723197888 (277h44m57.493839223s) [upload=222113]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 4 22:55:22.723203449 (277h44m57.493844784s) [upload=222113]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 4 22:55:23.013288575 (277h44m57.78392991s) [upload=222113]: Done uploading segments... Jul 4 22:55:23.013305126 (277h44m57.783946461s) [upload=222113]: Upload finished. err: <nil> Jul 4 22:56:14.040729919 (277h45m48.811371254s) [upload=222123]: Starting upload Jul 4 22:56:14.040785184 (277h45m48.81142652s) [upload=222123]: Done starting upload Jul 4 22:56:14.040880067 (277h45m48.811521412s) [upload=222123]: Uploading segments... Jul 4 22:56:14.040952875 (277h45m48.81159421s) [upload=222123]: Got next segment. Inline: false Jul 4 22:56:14.040988043 (277h45m48.811629378s) [upload=222123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 4 22:56:14.040993954 (277h45m48.811635289s) [upload=222123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 4 22:56:14.271530389 (277h45m49.042171725s) [seg_pos={0 0}, upload=222123]: Begin upload segment... Jul 4 22:56:14.271576417 (277h45m49.042217742s) [seg_pos={0 0}, upload=222123]: Done begin upload segment. Jul 4 22:56:14.271592628 (277h45m49.042233963s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271596175 (277h45m49.0422375s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.271966233 (277h45m49.042607568s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.272847128 (277h45m49.043488463s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273155167 (277h45m49.043796502s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273396249 (277h45m49.044037584s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273611220 (277h45m49.044252555s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.273860777 (277h45m49.044502113s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274079987 (277h45m49.044721312s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.274650668 (277h45m49.045292003s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275361208 (277h45m49.046002543s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.275903365 (277h45m49.0465447s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276584367 (277h45m49.047225703s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.276892256 (277h45m49.047533591s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277101517 (277h45m49.047742842s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:14.277557870 (277h45m49.048199215s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Uploading piece... Jul 4 22:56:15.875100346 (277h45m50.645741681s) [upload=222123]: Next returned nil segment Jul 4 22:56:15.875118811 (277h45m50.645760147s) [upload=222123]: Waiting for error group managing segments... Jul 4 22:56:16.162716458 (277h45m50.933357793s) [noise=true , address=104.36.82.11:4114 , piece_id=KXUXM2IPC2B7M7H5ZDKK3YK43PUXVJLWHNEET6BYWCEHJKTIMLSA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.217850705 (277h45m50.98849203s) [noise=true , address=104.36.82.8:4970 , piece_id=W5ERZV6LFUMHGTVBKAO4XMLZI7PEAR2AXWOHICC4WJPFEB23N32A, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.221479066 (277h45m50.992120421s) [noise=true , address=104.36.82.7:4910 , piece_id=TATGPLLJFM7KKLGC4IAKHYOXA7Q6AYNVAWFDCJFY5UOC5GA5XEXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.263761024 (277h45m51.034402359s) [noise=true , address=104.36.82.16:4293 , piece_id=B32YH2KGDXRK6UX3CVNPV2WZV2JV2WWVYFSY644DP3F3G5ZF3WHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.309476152 (277h45m51.080117487s) [noise=true , address=104.36.82.9:4007 , piece_id=MC5ZP4GFQCXAJKPGIINNC7PBNKUKNOJNVARQQWY2XAL4WB2ZRFXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.312243125 (277h45m51.08288445s) [noise=true , address=104.36.82.5:4838 , piece_id=NJZE7DGU3NLAYEGB2QOHWU6HRFBHDXYNNPGDMKW6UJTSLYE2RTNQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.353719753 (277h45m51.124361088s) [noise=true , address=104.36.82.6:4875 , piece_id=P4LJ5KFIL44VF3CT7XHMF6FA3HVZEDHNPONMLGJ2FN2I6O7GBOCQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.373738721 (277h45m51.144380046s) [noise=true , address=104.36.82.15:4257 , piece_id=MCECOYALODRGRZ724BSUNIZPNFHZKGHYBNPVZJEJOXPALQTMMYHA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.946599148 (277h45m51.717240473s) [noise=true , address=45.91.116.14:4190 , piece_id=DFNY7XXFUQJNKRZ55RVS2FJFKRAULLI36VJG627GTFPHOIGVJLAQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:16.990324278 (277h45m51.760965603s) [noise=true , address=45.91.116.9:4262 , piece_id=MNSBCBGCDF5QS5N43YN7P5RN5AYZEA4H4WN2X7DD2FDCLNK6ZKMA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:17.223786039 (277h45m51.994427374s) [noise=true , address=45.91.116.6:4974 , piece_id=YCJI6UOC4M3EBPOIDT4QFOMCLGCIK7535AHPFAEWKYKGIHL7ZCXQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: <nil> Jul 4 22:56:43.395328425 (277h46m18.16596975s) [noise=true , address=77.90.31.12:4589 , piece_id=NUOXZ4CKUTIOOUTX5DPBQCE2B6Z6JZCGAQGZD2ZK3KPDZAPVA2AA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395373622 (277h46m18.166014957s) [noise=true , address=77.90.31.17:4771 , piece_id=D2LMT3Q6WB5HYBBZ5AW44RJLJK2Y2L4MI75NAMUGBVWINRFWXLTA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: read tcp 188.166.225.170:39080->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395501978 (277h46m18.166143303s) [noise=true , address=77.90.31.13:4628 , piece_id=EL5CZLN7FC4VKGN7WR7XQV7BZ2LQ64FR6HF2D24ABTENVBPTAOMQ, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:47956->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395510454 (277h46m18.166151779s) [noise=true , address=77.90.31.18:4806 , piece_id=LP4WP5SZ4PCIZU2D4LKZNUAD6T4BMKTJTE2MQ6FCGLJS6R4BEZAA, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 188.166.225.170:60628->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395531454 (277h46m18.166172789s) [noise=true , address=77.90.31.16:4734 , piece_id=5YY2UBDV3BR7UWNWVJKC6AYQP6QU4RD3U6K4YEOCIJH34VLAY22Q, seg_pos={0 0}, upload=222123]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 188.166.225.170:41166->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 4 22:56:43.395549839 (277h46m18.166191164s) [seg_pos={0 0}, upload=222123]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 4 22:56:43.642038542 (277h46m18.412679877s) [upload=222123]: Done uploading segments... Jul 4 22:56:43.642064582 (277h46m18.412705917s) [upload=222123]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.017940157s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.271407444s FAIL (2025-07-04 23:04:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.271429857s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)