Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [XX.X.X..X...........X.X..X.X..XX.X.X..XXX......X.X.......X..] duration: 25.559111159s trace: https://jaeger.dev.storj.io/trace/5d65dca9c1ba7b13 FAIL (2026-05-15 23:49:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.559131719s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.341409646s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:48:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 6.341434894s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [.XX.X.X..X...........X.X..X.X..XX.X.X..XXX......X.X.......X.] duration: 7.33197931s trace: https://jaeger.dev.storj.io/trace/5d65dca9c1ba7b13 FAIL (2026-05-15 23:48:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.332014257s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [..XX.X.X..X...........X.X..X.X..XX.X.X..XXX......X.X.......X] duration: 1m0.000958866s trace: https://jaeger.dev.storj.io/trace/5d65dca9c1ba7b13 FAIL (2026-05-15 23:48:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000969177s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.636549279s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:48:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 6.636573085s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [....X........X..............X.......X.......................] duration: 4.014994452s trace: https://jaeger.dev.storj.io/trace/b60e0bd34355396 FAIL (2026-05-15 23:48:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.015075065s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [...XX.X.X..X...........X.X..X.X..XX.X.X..XXX......X.X.......] duration: 37.332986007s trace: https://jaeger.dev.storj.io/trace/1e335be559a4c340 FAIL (2026-05-15 23:47:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 37.333000803s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.....X........X..............X.......X......................] duration: 4.241831391s trace: https://jaeger.dev.storj.io/trace/b60e0bd34355396 FAIL (2026-05-15 23:47:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.241842161s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.897501619s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:47:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 6.89752766s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [....XX.X.X..X...........X.X..X.X..XX.X.X..XXX......X.X......] duration: 43.991596239s trace: https://jaeger.dev.storj.io/trace/1e335be559a4c340 FAIL (2026-05-15 23:47:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 43.991618211s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 10.316743214s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:46:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 10.316766319s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [......X........X..............X.......X.....................] duration: 4.153408355s trace: https://jaeger.dev.storj.io/trace/b60e0bd34355396 FAIL (2026-05-15 23:46:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.153431359s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.005990966s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:46:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 6.006016757s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.....XX.X.X..X...........X.X..X.X..XX.X.X..XXX......X.X.....] duration: 11.385657083s trace: https://jaeger.dev.storj.io/trace/1e335be559a4c340 FAIL (2026-05-15 23:45:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.38567635s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.......X........X..............X.......X....................] duration: 3.786166745s trace: https://jaeger.dev.storj.io/trace/b60e0bd34355396 FAIL (2026-05-15 23:45:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.786205749s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.07741032s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:45:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 6.077433065s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [........X........X..............X.......X...................] duration: 3.869289005s trace: https://jaeger.dev.storj.io/trace/b60e0bd34355396 FAIL (2026-05-15 23:45:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.869313471s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [......XX.X.X..X...........X.X..X.X..XX.X.X..XXX......X.X....] duration: 18.496919975s trace: https://jaeger.dev.storj.io/trace/1e335be559a4c340 FAIL (2026-05-15 23:44:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.496942908s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.........X........X..............X.......X..................] duration: 3.975815513s trace: https://jaeger.dev.storj.io/trace/b60e0bd34355396 FAIL (2026-05-15 23:44:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.975825894s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.056965016s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:44:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 6.056986778s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..........X........X..............X.......X.................] duration: 3.84888888s trace: https://jaeger.dev.storj.io/trace/b60e0bd34355396 FAIL (2026-05-15 23:44:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.848912175s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.......XX.X.X..X...........X.X..X.X..XX.X.X..XXX......X.X...] duration: 43.180059968s trace: https://jaeger.dev.storj.io/trace/1e335be559a4c340 FAIL (2026-05-15 23:44:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 43.180077883s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.758417478s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:43:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 5.758449288s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...........X........X..............X.......X................] duration: 3.938762798s trace: https://jaeger.dev.storj.io/trace/b60e0bd34355396 FAIL (2026-05-15 23:43:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.938815278s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 8.396006996s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:42:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 8.396032594s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [............X........X..............X.......X...............] duration: 4.246452652s trace: https://jaeger.dev.storj.io/trace/b60e0bd34355396 FAIL (2026-05-15 23:42:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.246479092s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [X.......XX.X.X..X...........X.X..X.X..XX.X.X..XXX......X.X..] duration: 10.932180005s trace: https://jaeger.dev.storj.io/trace/1e335be559a4c340 FAIL (2026-05-15 23:42:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.932197719s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [XX.......XX.X.X..X...........X.X..X.X..XX.X.X..XXX......X.X.] duration: 59.477963578s trace: https://jaeger.dev.storj.io/trace/1e335be559a4c340 FAIL (2026-05-15 23:42:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 59.477989566s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.............X........X..............X.......X..............] duration: 3.944688316s trace: https://jaeger.dev.storj.io/trace/b60e0bd34355396 FAIL (2026-05-15 23:42:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.944710037s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..............X........X..............X.......X.............] duration: 4.076156422s trace: https://jaeger.dev.storj.io/trace/b60e0bd34355396 FAIL (2026-05-15 23:41:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.076179707s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 10.985209977s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:41:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 10.985230036s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.3902272s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:41:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 6.390555577s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [.XX.......XX.X.X..X...........X.X..X.X..XX.X.X..XXX......X.X] duration: 1m0.00115205s trace: https://jaeger.dev.storj.io/trace/1e335be559a4c340 FAIL (2026-05-15 23:41:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001176826s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...............X........X..............X.......X............] duration: 3.894993978s trace: https://jaeger.dev.storj.io/trace/b60e0bd34355396 FAIL (2026-05-15 23:40:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.895021309s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.045271961s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:40:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 6.045296698s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [................X........X..............X.......X...........] duration: 4.882670869s trace: https://jaeger.dev.storj.io/trace/b60e0bd34355396 FAIL (2026-05-15 23:40:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.882703681s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [X.XX.......XX.X.X..X...........X.X..X.X..XX.X.X..XXX......X.] duration: 57.447985876s trace: https://jaeger.dev.storj.io/trace/4f1c034340642868 FAIL (2026-05-15 23:40:03 UTC) orbiter warning: context deadline exceeded Duration: 57.448002439s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 8.869564125s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:39:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 8.869588582s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [.X.XX.......XX.X.X..X...........X.X..X.X..XX.X.X..XXX......X] duration: 1m0.001569332s trace: https://jaeger.dev.storj.io/trace/4f1c034340642868 FAIL (2026-05-15 23:39:06 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001577398s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.................X........X..............X.......X..........] duration: 3.862788143s trace: https://jaeger.dev.storj.io/trace/b60e0bd34355396 FAIL (2026-05-15 23:38:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.862813792s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 11.23309821s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:38:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 11.233121714s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..................X........X..............X.......X.........] duration: 3.940360107s trace: https://jaeger.dev.storj.io/trace/b60e0bd34355396 FAIL (2026-05-15 23:38:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.940382871s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.459192007s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:38:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 6.459208768s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...................X........X..............X.......X........] duration: 5.000771846s trace: https://jaeger.dev.storj.io/trace/b60e0bd34355396 FAIL (2026-05-15 23:38:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000801812s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 11.070379306s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:37:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 11.070396729s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 13.446276682s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:37:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 13.446329914s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [X.X.XX.......XX.X.X..X...........X.X..X.X..XX.X.X..XXX......] duration: 8.023746766s trace: https://jaeger.dev.storj.io/trace/48dd39e5fafa937f FAIL (2026-05-15 23:37:07 UTC) orbiter warning: context deadline exceeded Duration: 8.023768006s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [....................X........X..............X.......X.......] duration: 3.928483913s trace: https://jaeger.dev.storj.io/trace/b60e0bd34355396 FAIL (2026-05-15 23:37:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.928510464s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.....................X........X..............X.......X......] duration: 3.680107422s trace: https://jaeger.dev.storj.io/trace/b60e0bd34355396 FAIL (2026-05-15 23:36:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.680121548s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 17.173291881s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:36:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 17.173314675s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)