Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 851586 reports (Page 2003 of 17032)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [XXX......X.X.......X.....X........X.X..X...X................] duration: 8.147186907s trace: https://jaeger.dev.storj.io/trace/3d8c777eda65ff41 FAIL (2026-05-16 00:19:31 UTC) orbiter warning: context deadline exceeded Duration: 8.147209288s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........................X...X....................X........] duration: 4.031154326s trace: https://jaeger.dev.storj.io/trace/ce745bfd19d7b1 FAIL (2026-05-16 00:19:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.031177321s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 194.091935ms trace: https://jaeger.dev.storj.io/trace/5f242b9ee760764b FAIL (2026-05-16 00:19:00 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 194.116863ms core uplink ap1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.XXX......X.X.......X.....X........X.X..X...X...............] duration: 7.49455235s trace: https://jaeger.dev.storj.io/trace/3d8c777eda65ff41 FAIL (2026-05-16 00:18:58 UTC) orbiter warning: context deadline exceeded Duration: 7.494576455s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........................X...X....................X.......] duration: 4.069177462s trace: https://jaeger.dev.storj.io/trace/ce745bfd19d7b1 FAIL (2026-05-16 00:18:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.06920716s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 10.528474347s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-16 00:18:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 10.528497651s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 187.899019ms trace: https://jaeger.dev.storj.io/trace/5f242b9ee760764b FAIL (2026-05-16 00:18:05 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 187.929428ms core uplink ap1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............................X...X....................X......] duration: 3.911248462s trace: https://jaeger.dev.storj.io/trace/ce745bfd19d7b1 FAIL (2026-05-16 00:17:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.91127397s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 9.692827633s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-16 00:17:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 9.692856887s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 180.763864ms trace: https://jaeger.dev.storj.io/trace/5f242b9ee760764b FAIL (2026-05-16 00:17:44 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 180.782229ms core uplink ap1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [..XXX......X.X.......X.....X........X.X..X...X..............] duration: 7.933608335s trace: https://jaeger.dev.storj.io/trace/3d8c777eda65ff41 FAIL (2026-05-16 00:17:43 UTC) orbiter warning: context deadline exceeded Duration: 7.933634316s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [X..XXX......X.X.......X.....X........X.X..X...X.............] duration: 8.029199966s trace: https://jaeger.dev.storj.io/trace/3d8c777eda65ff41 FAIL (2026-05-16 00:17:22 UTC) orbiter warning: context deadline exceeded Duration: 8.029221558s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 7.573630175s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-16 00:17:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 7.57364398s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.X..XXX......X.X.......X.....X........X.X..X...X............] duration: 7.58776036s trace: https://jaeger.dev.storj.io/trace/3d8c777eda65ff41 FAIL (2026-05-16 00:17:05 UTC) orbiter warning: context deadline exceeded Duration: 7.587783533s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 189.856173ms trace: https://jaeger.dev.storj.io/trace/5f242b9ee760764b FAIL (2026-05-16 00:17:02 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 189.882514ms core uplink ap1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............................X...X....................X.....] duration: 3.780811212s trace: https://jaeger.dev.storj.io/trace/ce745bfd19d7b1 FAIL (2026-05-16 00:16:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.780837883s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 8.597126886s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-16 00:16:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 8.597151602s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 201.961864ms trace: https://jaeger.dev.storj.io/trace/5f242b9ee760764b FAIL (2026-05-16 00:16:46 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 201.989928ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 9.394021148s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-16 00:16:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 9.39404338s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [X.X..XXX......X.X.......X.....X........X.X..X...X...........] duration: 7.791046991s trace: https://jaeger.dev.storj.io/trace/3d8c777eda65ff41 FAIL (2026-05-16 00:16:12 UTC) orbiter warning: context deadline exceeded Duration: 7.79107278s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 401.296735ms trace: https://jaeger.dev.storj.io/trace/5f242b9ee760764b FAIL (2026-05-16 00:16:00 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 401.351169ms core uplink ap1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............................X...X....................X....] duration: 3.793833008s trace: https://jaeger.dev.storj.io/trace/ce745bfd19d7b1 FAIL (2026-05-16 00:15:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.793874176s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 7.173131393s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-16 00:15:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 7.173157523s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [X..............................X...X....................X...] duration: 4.325339299s trace: https://jaeger.dev.storj.io/trace/ce745bfd19d7b1 FAIL (2026-05-16 00:15:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.325365361s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.X.X..XXX......X.X.......X.....X........X.X..X...X..........] duration: 7.311351877s trace: https://jaeger.dev.storj.io/trace/3d8c777eda65ff41 FAIL (2026-05-16 00:15:09 UTC) orbiter warning: context deadline exceeded Duration: 7.311424486s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 181.316629ms trace: https://jaeger.dev.storj.io/trace/5f242b9ee760764b FAIL (2026-05-16 00:14:48 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 181.341847ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.758570616s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-16 00:14:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 6.758597696s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 9.845406812s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-16 00:14:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 9.845430568s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001558699s trace: https://jaeger.dev.storj.io/trace/5f242b9ee760764b FAIL (2026-05-16 00:14:03 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 30.001582185s core uplink ap1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.X..............................X...X....................X..] duration: 3.682296846s trace: https://jaeger.dev.storj.io/trace/ce745bfd19d7b1 FAIL (2026-05-16 00:14:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.682312075s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [X.X.X..XXX......X.X.......X.....X........X.X..X...X.........] duration: 8.519165577s trace: https://jaeger.dev.storj.io/trace/3d8c777eda65ff41 FAIL (2026-05-16 00:14:02 UTC) orbiter warning: context deadline exceeded Duration: 8.519179603s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [XX.X.X..XXX......X.X.......X.....X........X.X..X...X........] duration: 8.903850334s trace: https://jaeger.dev.storj.io/trace/3d8c777eda65ff41 FAIL (2026-05-16 00:13:31 UTC) orbiter warning: context deadline exceeded Duration: 8.903880241s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.XX.X.X..XXX......X.X.......X.....X........X.X..X...X.......] duration: 7.821231328s trace: https://jaeger.dev.storj.io/trace/3d8c777eda65ff41 FAIL (2026-05-16 00:13:12 UTC) orbiter warning: context deadline exceeded Duration: 7.821264912s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..X..............................X...X....................X.] duration: 3.812814549s trace: https://jaeger.dev.storj.io/trace/ce745bfd19d7b1 FAIL (2026-05-16 00:13:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.812836852s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 11.724972116s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-16 00:13:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 11.724994447s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...X..............................X...X....................X] duration: 5.000802681s trace: https://jaeger.dev.storj.io/trace/ce745bfd19d7b1 FAIL (2026-05-16 00:12:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000823361s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [..XX.X.X..XXX......X.X.......X.....X........X.X..X...X......] duration: 7.587255396s trace: https://jaeger.dev.storj.io/trace/3d8c777eda65ff41 FAIL (2026-05-16 00:12:31 UTC) orbiter warning: context deadline exceeded Duration: 7.587277608s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....X..............................X...X....................] duration: 3.92999823s trace: https://jaeger.dev.storj.io/trace/34b4425d7c2a9299 FAIL (2026-05-16 00:12:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.930008439s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 7.437428426s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-16 00:12:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 7.437447913s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....X..............................X...X...................] duration: 4.323047627s trace: https://jaeger.dev.storj.io/trace/34b4425d7c2a9299 FAIL (2026-05-16 00:11:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.323109014s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [X..XX.X.X..XXX......X.X.......X.....X........X.X..X...X.....] duration: 7.708630904s trace: https://jaeger.dev.storj.io/trace/3d8c777eda65ff41 FAIL (2026-05-16 00:11:39 UTC) orbiter warning: context deadline exceeded Duration: 7.708655439s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 13.686408663s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-16 00:11:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 13.686435354s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [.X..XX.X.X..XXX......X.X.......X.....X........X.X..X...X....] duration: 10.388070642s trace: https://jaeger.dev.storj.io/trace/3d8c777eda65ff41 FAIL (2026-05-16 00:10:38 UTC) orbiter warning: context deadline exceeded Duration: 10.388097924s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......X..............................X...X..................] duration: 4.029875528s trace: https://jaeger.dev.storj.io/trace/34b4425d7c2a9299 FAIL (2026-05-16 00:10:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.029901085s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.980350427s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-16 00:09:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 6.980376317s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [X.X..XX.X.X..XXX......X.X.......X.....X........X.X..X...X...] duration: 12.540473794s trace: https://jaeger.dev.storj.io/trace/3d8c777eda65ff41 FAIL (2026-05-16 00:09:46 UTC) orbiter warning: context deadline exceeded Duration: 12.540485416s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [.X.X..XX.X.X..XXX......X.X.......X.....X........X.X..X...X..] duration: 48.48609061s trace: https://jaeger.dev.storj.io/trace/3d8c777eda65ff41 FAIL (2026-05-16 00:09:33 UTC) orbiter warning: context deadline exceeded Duration: 48.486104275s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......X..............................X...X.................] duration: 4.016680217s trace: https://jaeger.dev.storj.io/trace/34b4425d7c2a9299 FAIL (2026-05-16 00:09:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.016698552s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [X.......X..............................X...X................] duration: 3.944237582s trace: https://jaeger.dev.storj.io/trace/34b4425d7c2a9299 FAIL (2026-05-16 00:09:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.944284302s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 7.154076181s trace: https://jaeger.dev.storj.io/trace/7aee1db2369a7212 FAIL (2026-05-16 00:08:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments... May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment... May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment. May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece... May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments... May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil> May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments... May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil> Duration: 7.154100658s core uplink us-select-1

1
...
2001
2002
2003
2004
2005
...
17032