Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..............X.......X..............................X...X..] duration: 3.709393763s trace: https://jaeger.dev.storj.io/trace/34b4425d7c2a9299 FAIL (2026-05-15 23:58:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.709420404s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 11.242100423s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:58:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 11.242125891s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.04179915s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:58:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 6.041816212s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [X..............X.......X..............................X...X.] duration: 3.708859604s trace: https://jaeger.dev.storj.io/trace/34b4425d7c2a9299 FAIL (2026-05-15 23:57:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.708885334s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.........X.X..X.X..XX.X.X..XXX......X.X.......X.....X.......] duration: 7.72309348s trace: https://jaeger.dev.storj.io/trace/6e8031654c9547bc FAIL (2026-05-15 23:57:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.723116213s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 13.89369815s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:57:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 13.893720323s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [..........X.X..X.X..XX.X.X..XXX......X.X.......X.....X......] duration: 7.701444613s trace: https://jaeger.dev.storj.io/trace/6e8031654c9547bc FAIL (2026-05-15 23:57:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.701465763s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.X..............X.......X..............................X...X] duration: 5.000813612s trace: https://jaeger.dev.storj.io/trace/34b4425d7c2a9299 FAIL (2026-05-15 23:56:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00083355s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [...........X.X..X.X..XX.X.X..XXX......X.X.......X.....X.....] duration: 8.532799796s trace: https://jaeger.dev.storj.io/trace/6e8031654c9547bc FAIL (2026-05-15 23:56:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.532827408s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 9.181395776s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:56:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 9.181425893s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..X..............X.......X..............................X...] duration: 3.674563902s trace: https://jaeger.dev.storj.io/trace/75223f730a93402f FAIL (2026-05-15 23:56:00 UTC) orbiter warning: context deadline exceeded Duration: 3.674590862s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [X...........X.X..X.X..XX.X.X..XXX......X.X.......X.....X....] duration: 7.743551331s trace: https://jaeger.dev.storj.io/trace/6e8031654c9547bc FAIL (2026-05-15 23:55:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.743573014s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 9.844535168s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:55:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 9.844560766s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...X..............X.......X..............................X..] duration: 3.811503391s trace: https://jaeger.dev.storj.io/trace/75223f730a93402f FAIL (2026-05-15 23:54:58 UTC) orbiter warning: context deadline exceeded Duration: 3.8115446s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.422917257s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:54:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 6.422940211s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [.X...........X.X..X.X..XX.X.X..XXX......X.X.......X.....X...] duration: 7.465546433s trace: https://jaeger.dev.storj.io/trace/6e8031654c9547bc FAIL (2026-05-15 23:54:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.46556056s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.804123306s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:54:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 5.804146571s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [..X...........X.X..X.X..XX.X.X..XXX......X.X.......X.....X..] duration: 51.503832655s trace: https://jaeger.dev.storj.io/trace/6e8031654c9547bc FAIL (2026-05-15 23:54:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 51.503853604s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [....X..............X.......X..............................X.] duration: 3.729092723s trace: https://jaeger.dev.storj.io/trace/75223f730a93402f FAIL (2026-05-15 23:54:21 UTC) orbiter warning: context deadline exceeded Duration: 3.729119053s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [X..X...........X.X..X.X..XX.X.X..XXX......X.X.......X.....X.] duration: 17.80197981s trace: https://jaeger.dev.storj.io/trace/6e8031654c9547bc FAIL (2026-05-15 23:53:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.802030628s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 8.730151682s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:53:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 8.73016676s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [.X..X...........X.X..X.X..XX.X.X..XXX......X.X.......X.....X] duration: 1m0.001491421s trace: https://jaeger.dev.storj.io/trace/6e8031654c9547bc FAIL (2026-05-15 23:53:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001507514s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.....X..............X.......X..............................X] duration: 5.001051927s trace: https://jaeger.dev.storj.io/trace/75223f730a93402f FAIL (2026-05-15 23:53:11 UTC) orbiter warning: context deadline exceeded Duration: 5.001064311s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......X..............X.......X..............................] duration: 3.803330984s trace: https://jaeger.dev.storj.io/trace/b60e0bd34355396 FAIL (2026-05-15 23:52:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.803356523s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 9.262319375s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:52:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 9.262334204s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......X..............X.......X.............................] duration: 3.884087554s trace: https://jaeger.dev.storj.io/trace/b60e0bd34355396 FAIL (2026-05-15 23:52:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.88411619s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........X..............X.......X............................] duration: 4.055099769s trace: https://jaeger.dev.storj.io/trace/b60e0bd34355396 FAIL (2026-05-15 23:51:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.055128955s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 7.36056149s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:51:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 7.360585536s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [X.X..X...........X.X..X.X..XX.X.X..XXX......X.X.......X.....] duration: 19.332413305s trace: https://jaeger.dev.storj.io/trace/5d65dca9c1ba7b13 FAIL (2026-05-15 23:51:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.33244815s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [X........X..............X.......X...........................] duration: 3.697031566s trace: https://jaeger.dev.storj.io/trace/b60e0bd34355396 FAIL (2026-05-15 23:51:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.697057164s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [.X.X..X...........X.X..X.X..XX.X.X..XXX......X.X.......X....] duration: 37.515647047s trace: https://jaeger.dev.storj.io/trace/5d65dca9c1ba7b13 FAIL (2026-05-15 23:51:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 37.515694527s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.902025903s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:50:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 6.902048866s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.X........X..............X.......X..........................] duration: 3.741082333s trace: https://jaeger.dev.storj.io/trace/b60e0bd34355396 FAIL (2026-05-15 23:50:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.741107951s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [X.X.X..X...........X.X..X.X..XX.X.X..XXX......X.X.......X...] duration: 10.257133369s trace: https://jaeger.dev.storj.io/trace/5d65dca9c1ba7b13 FAIL (2026-05-15 23:50:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.257163086s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..X........X..............X.......X.........................] duration: 3.869932396s trace: https://jaeger.dev.storj.io/trace/b60e0bd34355396 FAIL (2026-05-15 23:49:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.869980618s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.787488794s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:49:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 5.787514733s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...X........X..............X.......X........................] duration: 3.790626633s trace: https://jaeger.dev.storj.io/trace/b60e0bd34355396 FAIL (2026-05-15 23:49:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.790650219s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [XX.X.X..X...........X.X..X.X..XX.X.X..XXX......X.X.......X..] duration: 25.559111159s trace: https://jaeger.dev.storj.io/trace/5d65dca9c1ba7b13 FAIL (2026-05-15 23:49:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.559131719s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.341409646s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:48:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 6.341434894s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [.XX.X.X..X...........X.X..X.X..XX.X.X..XXX......X.X.......X.] duration: 7.33197931s trace: https://jaeger.dev.storj.io/trace/5d65dca9c1ba7b13 FAIL (2026-05-15 23:48:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.332014257s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [..XX.X.X..X...........X.X..X.X..XX.X.X..XXX......X.X.......X] duration: 1m0.000958866s trace: https://jaeger.dev.storj.io/trace/5d65dca9c1ba7b13 FAIL (2026-05-15 23:48:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000969177s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.636549279s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:48:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 6.636573085s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [....X........X..............X.......X.......................] duration: 4.014994452s trace: https://jaeger.dev.storj.io/trace/b60e0bd34355396 FAIL (2026-05-15 23:48:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.015075065s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [...XX.X.X..X...........X.X..X.X..XX.X.X..XXX......X.X.......] duration: 37.332986007s trace: https://jaeger.dev.storj.io/trace/1e335be559a4c340 FAIL (2026-05-15 23:47:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 37.333000803s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.....X........X..............X.......X......................] duration: 4.241831391s trace: https://jaeger.dev.storj.io/trace/b60e0bd34355396 FAIL (2026-05-15 23:47:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.241842161s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.897501619s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:47:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 6.89752766s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [....XX.X.X..X...........X.X..X.X..XX.X.X..XXX......X.X......] duration: 43.991596239s trace: https://jaeger.dev.storj.io/trace/1e335be559a4c340 FAIL (2026-05-15 23:47:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 43.991618211s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 10.316743214s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:46:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 10.316766319s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [......X........X..............X.......X.....................] duration: 4.153408355s trace: https://jaeger.dev.storj.io/trace/b60e0bd34355396 FAIL (2026-05-15 23:46:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.153431359s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.005990966s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-15 23:46:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 6.006016757s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)