Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................................X....X................] duration: 8.422733717s FAIL (2025-07-09 17:12:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.422748757s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.749651475s FAIL (2025-07-09 17:12:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.749673196s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [XX.........................X........................X.......] duration: 7.440975758s FAIL (2025-07-09 17:12:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.441011567s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.XX.........................X........................X......] duration: 10.710997466s FAIL (2025-07-09 17:11:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.711017786s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.96126723s FAIL (2025-07-09 17:11:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.961290245s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X....X...............] duration: 5.652095245s FAIL (2025-07-09 17:11:10 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.652118971s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.991207423s FAIL (2025-07-09 17:10:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.991235788s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................................X....X..............] duration: 5.752858693s FAIL (2025-07-09 17:10:48 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.752893069s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..XX.........................X........................X.....] duration: 6.187643929s FAIL (2025-07-09 17:10:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.187665419s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X....X.............] duration: 24.144850209s FAIL (2025-07-09 17:10:33 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.144872872s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...XX.........................X........................X....] duration: 9.429368186s FAIL (2025-07-09 17:10:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.429388866s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.751499209s FAIL (2025-07-09 17:09:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.751526653s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X....X............] duration: 5.73385791s FAIL (2025-07-09 17:09:43 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.733914018s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.770136419s FAIL (2025-07-09 17:09:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.770164493s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X...XX.........................X........................X...] duration: 6.769821427s FAIL (2025-07-09 17:09:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.769846204s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X....X...........] duration: 5.588470165s FAIL (2025-07-09 17:09:03 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.588493881s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.615445684s FAIL (2025-07-09 17:08:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.615471534s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................................X....X..........] duration: 5.585816076s FAIL (2025-07-09 17:08:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.585829703s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [XX...XX.........................X........................X..] duration: 11.541340849s FAIL (2025-07-09 17:08:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.541363272s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X....X.........] duration: 36.489769837s FAIL (2025-07-09 17:08:41 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 36.489780057s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X....X........] duration: 39.243983647s FAIL (2025-07-09 17:08:04 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 39.244030105s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [XXX...XX.........................X........................X.] duration: 7.667788555s FAIL (2025-07-09 17:07:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.667818001s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.07487157s FAIL (2025-07-09 17:07:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.074895417s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.XXX...XX.........................X........................X] duration: 30.000369819s FAIL (2025-07-09 17:07:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000394476s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X....X.......] duration: 5.590258617s FAIL (2025-07-09 17:07:00 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.590293554s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 7.926605604s FAIL (2025-07-09 17:06:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.926631295s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..XXX...XX.........................X........................] duration: 9.519819918s FAIL (2025-07-09 17:06:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.519842953s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X..XXX...XX.........................X.......................] duration: 22.551400492s FAIL (2025-07-09 17:06:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.551423947s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.218825418s FAIL (2025-07-09 17:06:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.218849724s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................................X....X......] duration: 12.280601474s FAIL (2025-07-09 17:06:03 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.280624358s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X..XXX...XX.........................X......................] duration: 12.544751111s FAIL (2025-07-09 17:05:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.544774806s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X..XXX...XX.........................X.....................] duration: 7.871825515s FAIL (2025-07-09 17:05:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.871851055s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 15.398008505s FAIL (2025-07-09 17:05:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 15.398034585s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X....X.....] duration: 5.581761618s FAIL (2025-07-09 17:05:09 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.581806544s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X....X....] duration: 37.591025362s FAIL (2025-07-09 17:04:49 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 37.591038807s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X..XXX...XX.........................X....................] duration: 7.650736072s FAIL (2025-07-09 17:04:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.650751231s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.784410863s FAIL (2025-07-09 17:04:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.78444103s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X....X...] duration: 43.978992741s FAIL (2025-07-09 17:04:11 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 43.979012799s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X..XXX...XX.........................X...................] duration: 8.141197424s FAIL (2025-07-09 17:03:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.141220067s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.780506932s FAIL (2025-07-09 17:03:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.780532772s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X....X..] duration: 5.539575683s FAIL (2025-07-09 17:02:59 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.539590361s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X....X..XXX...XX.........................X..................] duration: 20.16499192s FAIL (2025-07-09 17:02:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.165008952s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X....X.] duration: 6.117630264s FAIL (2025-07-09 17:02:30 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.117647357s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.808132237s FAIL (2025-07-09 17:02:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.808158527s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X....X] duration: 1m0.000882959s FAIL (2025-07-09 17:02:16 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000902626s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 10.182249028s FAIL (2025-07-09 17:01:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 10.182281883s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X....X..XXX...XX.........................X.................] duration: 10.828160451s FAIL (2025-07-09 17:01:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.828690255s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.988713485s FAIL (2025-07-09 17:01:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.988738543s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X....X..XXX...XX.........................X................] duration: 6.469895732s FAIL (2025-07-09 17:01:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.469948544s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.804591982s FAIL (2025-07-09 17:00:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.804615286s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)