Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.956259474s FAIL (2025-07-09 17:00:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.956290523s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X....X..XXX...XX.........................X...............] duration: 6.003846646s FAIL (2025-07-09 17:00:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.003871865s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.689202611s FAIL (2025-07-09 17:00:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.689221938s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 36.599302239s FAIL (2025-07-09 17:00:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 36.599326136s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X....X..XXX...XX.........................X..............] duration: 7.351208905s FAIL (2025-07-09 16:59:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.351230086s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.04688724s FAIL (2025-07-09 16:59:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.046907079s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....X....X..XXX...XX.........................X.............] duration: 10.110063905s FAIL (2025-07-09 16:58:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.110090095s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.495777099s FAIL (2025-07-09 16:58:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.495801445s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.676740331s FAIL (2025-07-09 16:58:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.676759169s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......X....X..XXX...XX.........................X............] duration: 9.413344332s FAIL (2025-07-09 16:58:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.413406461s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.887293803s FAIL (2025-07-09 16:58:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.887321727s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.67374553s FAIL (2025-07-09 16:58:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.673757694s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.000439289s FAIL (2025-07-09 16:57:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000460378s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......X....X..XXX...XX.........................X...........] duration: 9.832048498s FAIL (2025-07-09 16:57:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.832071173s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 7.53239783s FAIL (2025-07-09 16:57:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.532420835s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [........X....X..XXX...XX.........................X..........] duration: 6.116972306s FAIL (2025-07-09 16:56:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.117002955s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.963728286s FAIL (2025-07-09 16:56:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.963748525s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........X....X..XXX...XX.........................X.........] duration: 10.045462315s FAIL (2025-07-09 16:56:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.045488536s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..........X....X..XXX...XX.........................X........] duration: 9.306089675s FAIL (2025-07-09 16:55:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.306111668s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.272699086s FAIL (2025-07-09 16:55:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.272727731s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.777727177s FAIL (2025-07-09 16:54:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.777748558s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...........X....X..XXX...XX.........................X.......] duration: 11.243978296s FAIL (2025-07-09 16:54:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.244033431s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [............X....X..XXX...XX.........................X......] duration: 6.930116519s FAIL (2025-07-09 16:54:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.930138631s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.912428168s FAIL (2025-07-09 16:53:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.912454129s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.............X....X..XXX...XX.........................X.....] duration: 10.8202455s FAIL (2025-07-09 16:53:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.82026722s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..............X....X..XXX...XX.........................X....] duration: 9.975506988s FAIL (2025-07-09 16:52:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.975559709s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.801748851s FAIL (2025-07-09 16:52:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.801774199s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...............X....X..XXX...XX.........................X...] duration: 6.898773409s FAIL (2025-07-09 16:52:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.898793217s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.298948503s FAIL (2025-07-09 16:51:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.298973309s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [................X....X..XXX...XX.........................X..] duration: 5.933655638s FAIL (2025-07-09 16:51:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.933682068s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.043439791s FAIL (2025-07-09 16:50:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.043462636s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................X....X..XXX...XX.........................X.] duration: 16.251779116s FAIL (2025-07-09 16:50:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.251790988s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..................X....X..XXX...XX.........................X] duration: 30.000868145s FAIL (2025-07-09 16:50:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000893644s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 7.801819923s FAIL (2025-07-09 16:50:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.801847576s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.961525692s FAIL (2025-07-09 16:50:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.96155635s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.865005349s FAIL (2025-07-09 16:49:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.865032341s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...................X....X..XXX...XX.........................] duration: 6.435512461s FAIL (2025-07-09 16:49:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.435535545s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.891306929s FAIL (2025-07-09 16:48:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.891340655s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....................X....X..XXX...XX........................] duration: 12.019728725s FAIL (2025-07-09 16:48:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.019759514s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.745380007s FAIL (2025-07-09 16:48:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.745404965s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....................X....X..XXX...XX.......................] duration: 6.635777257s FAIL (2025-07-09 16:47:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.635798857s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.939201107s FAIL (2025-07-09 16:47:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.939220303s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......................X....X..XXX...XX......................] duration: 6.401064828s FAIL (2025-07-09 16:46:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.401091288s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.790475097s FAIL (2025-07-09 16:46:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.790512659s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......................X....X..XXX...XX.....................] duration: 10.51312421s FAIL (2025-07-09 16:46:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.513148325s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.819164957s FAIL (2025-07-09 16:45:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.819183493s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [........................X....X..XXX...XX....................] duration: 6.295120519s FAIL (2025-07-09 16:45:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.295145006s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 8.52253126s FAIL (2025-07-09 16:44:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.522561157s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.787259959s FAIL (2025-07-09 16:44:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.787279737s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........................X....X..XXX...XX...................] duration: 10.751692456s FAIL (2025-07-09 16:44:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.75171531s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)