Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X........................X.......................] duration: 6.272114311s FAIL (2025-07-09 17:21:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.272134118s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................X....X...........................] duration: 12.287597677s FAIL (2025-07-09 17:21:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.287626432s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X........................X......................] duration: 5.902784423s FAIL (2025-07-09 17:21:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.902813258s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X........................X.....................] duration: 6.337656703s FAIL (2025-07-09 17:21:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.337681561s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................X....X..........................] duration: 5.51866309s FAIL (2025-07-09 17:20:41 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.518686724s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X........................X....................] duration: 5.984435034s FAIL (2025-07-09 17:20:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.984454452s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................X....X.........................] duration: 32.179441754s FAIL (2025-07-09 17:20:05 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 32.179464518s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X........................X...................] duration: 6.092104542s FAIL (2025-07-09 17:19:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.092124s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................X....X........................] duration: 5.475692048s FAIL (2025-07-09 17:19:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.475712036s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X........................X..................] duration: 6.573555848s FAIL (2025-07-09 17:18:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.573576588s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................X....X.......................] duration: 5.521976603s FAIL (2025-07-09 17:17:59 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.521999756s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X........................X.................] duration: 5.913103256s FAIL (2025-07-09 17:17:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.913123985s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X........................X................] duration: 7.616520496s FAIL (2025-07-09 17:17:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.616649814s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................X....X......................] duration: 5.760614209s FAIL (2025-07-09 17:16:58 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.760632034s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X........................X...............] duration: 6.22677969s FAIL (2025-07-09 17:16:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.226801641s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.832746592s FAIL (2025-07-09 17:16:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.832776148s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.740059967s FAIL (2025-07-09 17:16:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.740084173s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................X....X.....................] duration: 25.889644885s FAIL (2025-07-09 17:16:27 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.889671697s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.853300111s FAIL (2025-07-09 17:15:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.853321673s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X........................X..............] duration: 6.025539816s FAIL (2025-07-09 17:15:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.025565635s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.820004084s FAIL (2025-07-09 17:15:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.820030553s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X........................X.............] duration: 6.140590278s FAIL (2025-07-09 17:15:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.140615866s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................X....X....................] duration: 5.496642198s FAIL (2025-07-09 17:15:07 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.496753171s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X........................X............] duration: 6.321836999s FAIL (2025-07-09 17:14:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.321865364s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X........................X...........] duration: 5.930746901s FAIL (2025-07-09 17:14:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.930771408s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................X....X...................] duration: 20.590431674s FAIL (2025-07-09 17:14:40 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.590453065s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.832119922s FAIL (2025-07-09 17:14:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.832139179s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X........................X..........] duration: 6.035940043s FAIL (2025-07-09 17:13:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.035962066s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.02921719s FAIL (2025-07-09 17:13:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.029250454s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................X....X..................] duration: 5.542530108s FAIL (2025-07-09 17:13:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.542575857s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X....X.................] duration: 5.698492018s FAIL (2025-07-09 17:12:49 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.698517206s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X........................X.........] duration: 8.041522023s FAIL (2025-07-09 17:12:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.041542611s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.........................X........................X........] duration: 9.934955801s FAIL (2025-07-09 17:12:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.934975479s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................................X....X................] duration: 8.422733717s FAIL (2025-07-09 17:12:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.422748757s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.749651475s FAIL (2025-07-09 17:12:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.749673196s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [XX.........................X........................X.......] duration: 7.440975758s FAIL (2025-07-09 17:12:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.441011567s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.XX.........................X........................X......] duration: 10.710997466s FAIL (2025-07-09 17:11:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.711017786s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.96126723s FAIL (2025-07-09 17:11:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.961290245s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X....X...............] duration: 5.652095245s FAIL (2025-07-09 17:11:10 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.652118971s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.991207423s FAIL (2025-07-09 17:10:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.991235788s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................................X....X..............] duration: 5.752858693s FAIL (2025-07-09 17:10:48 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.752893069s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..XX.........................X........................X.....] duration: 6.187643929s FAIL (2025-07-09 17:10:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.187665419s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X....X.............] duration: 24.144850209s FAIL (2025-07-09 17:10:33 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.144872872s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...XX.........................X........................X....] duration: 9.429368186s FAIL (2025-07-09 17:10:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.429388866s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.751499209s FAIL (2025-07-09 17:09:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.751526653s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X....X............] duration: 5.73385791s FAIL (2025-07-09 17:09:43 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.733914018s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.770136419s FAIL (2025-07-09 17:09:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.770164493s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X...XX.........................X........................X...] duration: 6.769821427s FAIL (2025-07-09 17:09:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.769846204s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X....X...........] duration: 5.588470165s FAIL (2025-07-09 17:09:03 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.588493881s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.615445684s FAIL (2025-07-09 17:08:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 16:31:38.274172141 (391h21m13.044813476s) [upload=313111]: Done uploading segments... Jul 9 16:31:38.274189244 (391h21m13.044830579s) [upload=313111]: Upload finished. err: <nil> Jul 9 16:32:01.041247772 (391h21m35.811889107s) [upload=313117]: Starting upload Jul 9 16:32:01.041300824 (391h21m35.811942159s) [upload=313117]: Done starting upload Jul 9 16:32:01.041397017 (391h21m35.812038352s) [upload=313117]: Uploading segments... Jul 9 16:32:01.041531705 (391h21m35.81217304s) [upload=313117]: Got next segment. Inline: false Jul 9 16:32:01.041820527 (391h21m35.812461862s) [upload=313117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 16:32:01.041826830 (391h21m35.812468165s) [upload=313117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 16:32:01.275272217 (391h21m36.045913552s) [seg_pos={0 0}, upload=313117]: Begin upload segment... Jul 9 16:32:01.275326150 (391h21m36.045967485s) [seg_pos={0 0}, upload=313117]: Done begin upload segment. Jul 9 16:32:01.275350878 (391h21m36.045992213s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275367620 (391h21m36.046008955s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275834934 (391h21m36.046476279s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.275876563 (391h21m36.046517908s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276273243 (391h21m36.046914588s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276362754 (391h21m36.047004099s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276594647 (391h21m36.047235982s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.276822102 (391h21m36.047463447s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277103311 (391h21m36.047744646s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277490181 (391h21m36.048131526s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.277848246 (391h21m36.048489581s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278169540 (391h21m36.048810875s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278513028 (391h21m36.049154363s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.278842398 (391h21m36.049483743s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279029726 (391h21m36.049671061s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:01.279269344 (391h21m36.049910679s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Uploading piece... Jul 9 16:32:02.918073813 (391h21m37.688715148s) [upload=313117]: Next returned nil segment Jul 9 16:32:02.918089733 (391h21m37.688731068s) [upload=313117]: Waiting for error group managing segments... Jul 9 16:32:03.194073604 (391h21m37.964714939s) [noise=true , address=104.36.82.11:4115 , piece_id=HOWXIKR34WWKWW2DOGR7XOYXU3YHXC4UZXHOXKS4TZD3DII77MWA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.273979877 (391h21m38.044621212s) [noise=true , address=104.36.82.9:4006 , piece_id=D7WIJUTQBDKVGK6V63Z3K62ZZOC535DKV7BSHT4Q47KWTG46ITTA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.321204545 (391h21m38.09184588s) [noise=true , address=104.36.82.2:4730 , piece_id=JTV3JD2LBSPIJ5JU3HZ4MYS3FHTDYX5DE5WQIMCFQKI7RYT6XQ3A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.678104009 (391h21m38.448745344s) [noise=true , address=104.36.82.15:4259 , piece_id=QR3NKPOBGJLINC7IBFRC4FLNOWLFO3YBBQJOS5W5PXVZZKTU5FJQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.680245186 (391h21m38.450886521s) [noise=true , address=104.36.82.5:4840 , piece_id=R76WOXMNRHMKCWIO6EWMZN7TLUGYRJS2T4NPQ3NL6KU4XBPEE3RQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:03.800809634 (391h21m38.571450969s) [noise=true , address=104.36.82.13:4186 , piece_id=DV7NSPO4GITOTQEYRHXU7IXSWKIXZWSDXNWVNON2LT3E324SBWXQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.519843928 (391h21m39.290485263s) [noise=true , address=45.91.116.6:4974 , piece_id=YQ23TG5KXTKTPLDWXUQAWWKQRI5EDVOI53JT6MVO7GRTTHTNLQRA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.821747337 (391h21m39.592388672s) [noise=true , address=45.91.116.7:4010 , piece_id=QBMBXU2IFRW42XZNXW6SR36ISLLNROKFHDTQUJKAGTFJDAPOWCFA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:04.910297327 (391h21m39.680938672s) [noise=true , address=104.36.82.14:4223 , piece_id=XVRSM3TZ5FBZGX5RTWQRNBNYQ4V3RSKY3YF2I5K57RHVECVIF2DQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.150192765 (391h21m39.9208341s) [noise=true , address=104.36.82.12:4149 , piece_id=J6IK4HHS7XW6YMOA6S3EUWKGDCDGDVVV3K5ZAEXD7W6OYOT7GC5A, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:05.685714101 (391h21m40.456355436s) [noise=true , address=45.91.116.4:4937 , piece_id=2SJ2DWQCMC5SVB3FD6YPBP2D5WJOO5SCPGMUVHHJNXAQO364TGSQ, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:06.319904155 (391h21m41.09054549s) [noise=true , address=104.36.82.6:4875 , piece_id=PZYO2C6ULFMDHKT2TBG4C52SNHBOVR57R434BMUKHVWH5C226A6Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: <nil> Jul 9 16:32:30.399810430 (391h22m5.170451765s) [noise=true , address=77.90.31.12:4590 , piece_id=X5G6CVQKJAQYTXLWRINFVOX3SN3KAIEU5PNOUWOR2G5AJHE454TA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53308->77.90.31.12:4590: use of closed network connection Jul 9 16:32:30.399843403 (391h22m5.170484808s) [noise=true , address=77.90.31.16:4735 , piece_id=55KVC7SZ3Q2R6JUDGTHJISVVFOJ7P2VZQLKMMT2PMBUFNUE4SZBA, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47088->77.90.31.16:4735: use of closed network connection Jul 9 16:32:30.399913297 (391h22m5.170554672s) [noise=true , address=77.90.31.19:4841 , piece_id=QOFXTLOFKJWLFDNMIBVBKXC4QFUFYXVJVBL6GWGWSECJXFTIYC3Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42494->77.90.31.19:4841: use of closed network connection Jul 9 16:32:30.399983000 (391h22m5.170624395s) [noise=true , address=77.90.31.14:4662 , piece_id=AHZZVSMCQOJWLJIY77FYGT7OHOIBV6SSXFCZSHZE3T6HI2SNF46Q, seg_pos={0 0}, upload=313117]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34252->77.90.31.14:4662: use of closed network connection Jul 9 16:32:30.400030281 (391h22m5.170671696s) [seg_pos={0 0}, upload=313117]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 9 16:32:30.639087954 (391h22m5.4097293s) [upload=313117]: Done uploading segments... Jul 9 16:32:30.639116780 (391h22m5.409758115s) [upload=313117]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.615471534s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)