Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
401.296735ms trace:
https://jaeger.dev.storj.io/trace/5f242b9ee760764b
FAIL (2026-05-16 00:16:00 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 401.351169ms
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [..............................X...X....................X....] duration:
3.793833008s trace:
https://jaeger.dev.storj.io/trace/ce745bfd19d7b1
FAIL (2026-05-16 00:15:55 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.793874176s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
7.173131393s trace:
https://jaeger.dev.storj.io/trace/7aee1db2369a7212
FAIL (2026-05-16 00:15:23 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload
May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload
May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments...
May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false
May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment...
May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment.
May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment
May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments...
May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces
May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments...
May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil>
Duration: 7.173157523s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [X..............................X...X....................X...] duration:
4.325339299s trace:
https://jaeger.dev.storj.io/trace/ce745bfd19d7b1
FAIL (2026-05-16 00:15:15 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.325365361s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
13 out of 60 is failed (threshold=3) [.X.X..XXX......X.X.......X.....X........X.X..X...X..........] duration:
7.311351877s trace:
https://jaeger.dev.storj.io/trace/3d8c777eda65ff41
FAIL (2026-05-16 00:15:09 UTC)
orbiter warning: context deadline exceeded
Duration: 7.311424486s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
181.316629ms trace:
https://jaeger.dev.storj.io/trace/5f242b9ee760764b
FAIL (2026-05-16 00:14:48 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 181.341847ms
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
6.758570616s trace:
https://jaeger.dev.storj.io/trace/7aee1db2369a7212
FAIL (2026-05-16 00:14:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload
May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload
May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments...
May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false
May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment...
May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment.
May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment
May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments...
May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces
May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments...
May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil>
Duration: 6.758597696s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
9.845406812s trace:
https://jaeger.dev.storj.io/trace/7aee1db2369a7212
FAIL (2026-05-16 00:14:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload
May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload
May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments...
May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false
May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment...
May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment.
May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment
May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments...
May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces
May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments...
May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil>
Duration: 9.845430568s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.001558699s trace:
https://jaeger.dev.storj.io/trace/5f242b9ee760764b
FAIL (2026-05-16 00:14:03 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 30.001582185s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.X..............................X...X....................X..] duration:
3.682296846s trace:
https://jaeger.dev.storj.io/trace/ce745bfd19d7b1
FAIL (2026-05-16 00:14:03 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.682312075s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
14 out of 60 is failed (threshold=3) [X.X.X..XXX......X.X.......X.....X........X.X..X...X.........] duration:
8.519165577s trace:
https://jaeger.dev.storj.io/trace/3d8c777eda65ff41
FAIL (2026-05-16 00:14:02 UTC)
orbiter warning: context deadline exceeded
Duration: 8.519179603s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
15 out of 60 is failed (threshold=3) [XX.X.X..XXX......X.X.......X.....X........X.X..X...X........] duration:
8.903850334s trace:
https://jaeger.dev.storj.io/trace/3d8c777eda65ff41
FAIL (2026-05-16 00:13:31 UTC)
orbiter warning: context deadline exceeded
Duration: 8.903880241s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
15 out of 60 is failed (threshold=3) [.XX.X.X..XXX......X.X.......X.....X........X.X..X...X.......] duration:
7.821231328s trace:
https://jaeger.dev.storj.io/trace/3d8c777eda65ff41
FAIL (2026-05-16 00:13:12 UTC)
orbiter warning: context deadline exceeded
Duration: 7.821264912s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..X..............................X...X....................X.] duration:
3.812814549s trace:
https://jaeger.dev.storj.io/trace/ce745bfd19d7b1
FAIL (2026-05-16 00:13:11 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.812836852s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
11.724972116s trace:
https://jaeger.dev.storj.io/trace/7aee1db2369a7212
FAIL (2026-05-16 00:13:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload
May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload
May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments...
May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false
May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment...
May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment.
May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment
May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments...
May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces
May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments...
May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil>
Duration: 11.724994447s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...X..............................X...X....................X] duration:
5.000802681s trace:
https://jaeger.dev.storj.io/trace/ce745bfd19d7b1
FAIL (2026-05-16 00:12:41 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000823361s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
15 out of 60 is failed (threshold=3) [..XX.X.X..XXX......X.X.......X.....X........X.X..X...X......] duration:
7.587255396s trace:
https://jaeger.dev.storj.io/trace/3d8c777eda65ff41
FAIL (2026-05-16 00:12:31 UTC)
orbiter warning: context deadline exceeded
Duration: 7.587277608s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [....X..............................X...X....................] duration:
3.92999823s trace:
https://jaeger.dev.storj.io/trace/34b4425d7c2a9299
FAIL (2026-05-16 00:12:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.930008439s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
7.437428426s trace:
https://jaeger.dev.storj.io/trace/7aee1db2369a7212
FAIL (2026-05-16 00:12:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload
May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload
May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments...
May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false
May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment...
May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment.
May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment
May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments...
May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces
May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments...
May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil>
Duration: 7.437447913s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [.....X..............................X...X...................] duration:
4.323047627s trace:
https://jaeger.dev.storj.io/trace/34b4425d7c2a9299
FAIL (2026-05-16 00:11:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.323109014s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
16 out of 60 is failed (threshold=3) [X..XX.X.X..XXX......X.X.......X.....X........X.X..X...X.....] duration:
7.708630904s trace:
https://jaeger.dev.storj.io/trace/3d8c777eda65ff41
FAIL (2026-05-16 00:11:39 UTC)
orbiter warning: context deadline exceeded
Duration: 7.708655439s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
13.686408663s trace:
https://jaeger.dev.storj.io/trace/7aee1db2369a7212
FAIL (2026-05-16 00:11:08 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload
May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload
May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments...
May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false
May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment...
May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment.
May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment
May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments...
May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces
May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments...
May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil>
Duration: 13.686435354s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
16 out of 60 is failed (threshold=3) [.X..XX.X.X..XXX......X.X.......X.....X........X.X..X...X....] duration:
10.388070642s trace:
https://jaeger.dev.storj.io/trace/3d8c777eda65ff41
FAIL (2026-05-16 00:10:38 UTC)
orbiter warning: context deadline exceeded
Duration: 10.388097924s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [......X..............................X...X..................] duration:
4.029875528s trace:
https://jaeger.dev.storj.io/trace/34b4425d7c2a9299
FAIL (2026-05-16 00:10:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.029901085s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
6.980350427s trace:
https://jaeger.dev.storj.io/trace/7aee1db2369a7212
FAIL (2026-05-16 00:09:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload
May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload
May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments...
May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false
May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment...
May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment.
May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment
May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments...
May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces
May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments...
May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil>
Duration: 6.980376317s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
17 out of 60 is failed (threshold=3) [X.X..XX.X.X..XXX......X.X.......X.....X........X.X..X...X...] duration:
12.540473794s trace:
https://jaeger.dev.storj.io/trace/3d8c777eda65ff41
FAIL (2026-05-16 00:09:46 UTC)
orbiter warning: context deadline exceeded
Duration: 12.540485416s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
17 out of 60 is failed (threshold=3) [.X.X..XX.X.X..XXX......X.X.......X.....X........X.X..X...X..] duration:
48.48609061s trace:
https://jaeger.dev.storj.io/trace/3d8c777eda65ff41
FAIL (2026-05-16 00:09:33 UTC)
orbiter warning: context deadline exceeded
Duration: 48.486104275s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [.......X..............................X...X.................] duration:
4.016680217s trace:
https://jaeger.dev.storj.io/trace/34b4425d7c2a9299
FAIL (2026-05-16 00:09:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.016698552s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [X.......X..............................X...X................] duration:
3.944237582s trace:
https://jaeger.dev.storj.io/trace/34b4425d7c2a9299
FAIL (2026-05-16 00:09:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.944284302s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
7.154076181s trace:
https://jaeger.dev.storj.io/trace/7aee1db2369a7212
FAIL (2026-05-16 00:08:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload
May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload
May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments...
May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false
May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment...
May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment.
May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment
May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments...
May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces
May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments...
May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil>
Duration: 7.154100658s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
17 out of 60 is failed (threshold=3) [..X.X..XX.X.X..XXX......X.X.......X.....X........X.X..X...X.] duration:
25.360938936s trace:
https://jaeger.dev.storj.io/trace/3d8c777eda65ff41
FAIL (2026-05-16 00:08:45 UTC)
orbiter warning: context deadline exceeded
Duration: 25.360951168s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
7.681439339s trace:
https://jaeger.dev.storj.io/trace/7aee1db2369a7212
FAIL (2026-05-16 00:08:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload
May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload
May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments...
May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false
May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment...
May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment.
May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment
May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments...
May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces
May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments...
May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil>
Duration: 7.681463495s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [X..X.X..XX.X.X..XXX......X.X.......X.....X........X.X..X...X] duration:
1m0.001651009s trace:
https://jaeger.dev.storj.io/trace/3d8c777eda65ff41
FAIL (2026-05-16 00:08:19 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.00167175s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.X.......X..............................X...X...............] duration:
3.907774093s trace:
https://jaeger.dev.storj.io/trace/34b4425d7c2a9299
FAIL (2026-05-16 00:08:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.907847763s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..X.......X..............................X...X..............] duration:
3.807450003s trace:
https://jaeger.dev.storj.io/trace/34b4425d7c2a9299
FAIL (2026-05-16 00:07:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.807474419s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
6.502033804s trace:
https://jaeger.dev.storj.io/trace/7aee1db2369a7212
FAIL (2026-05-16 00:07:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload
May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload
May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments...
May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false
May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment...
May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment.
May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment
May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments...
May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces
May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments...
May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil>
Duration: 6.502059453s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...X.......X..............................X...X.............] duration:
4.047134369s trace:
https://jaeger.dev.storj.io/trace/34b4425d7c2a9299
FAIL (2026-05-16 00:07:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.047160097s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
6.957761162s trace:
https://jaeger.dev.storj.io/trace/7aee1db2369a7212
FAIL (2026-05-16 00:07:07 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload
May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload
May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments...
May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false
May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment...
May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment.
May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment
May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments...
May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces
May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments...
May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil>
Duration: 6.957788715s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
17 out of 60 is failed (threshold=3) [.X..X.X..XX.X.X..XXX......X.X.......X.....X........X.X..X...] duration:
10.167823923s trace:
https://jaeger.dev.storj.io/trace/6f01b97f3ddb54a7
FAIL (2026-05-16 00:06:50 UTC)
orbiter warning: context deadline exceeded
Duration: 10.167838069s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [X.X..X.X..XX.X.X..XXX......X.X.......X.....X........X.X..X..] duration:
52.457478654s trace:
https://jaeger.dev.storj.io/trace/6f01b97f3ddb54a7
FAIL (2026-05-16 00:06:40 UTC)
orbiter warning: context deadline exceeded
Duration: 52.45750811s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
7.495812827s trace:
https://jaeger.dev.storj.io/trace/7aee1db2369a7212
FAIL (2026-05-16 00:06:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload
May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload
May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments...
May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false
May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment...
May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment.
May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment
May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments...
May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces
May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments...
May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil>
Duration: 7.49583593s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [....X.......X..............................X...X............] duration:
3.828814931s trace:
https://jaeger.dev.storj.io/trace/34b4425d7c2a9299
FAIL (2026-05-16 00:06:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.828834549s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
6.027618212s trace:
https://jaeger.dev.storj.io/trace/7aee1db2369a7212
FAIL (2026-05-16 00:05:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload
May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload
May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments...
May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false
May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment...
May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment.
May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment
May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments...
May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces
May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments...
May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil>
Duration: 6.027641307s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [.X.X..X.X..XX.X.X..XXX......X.X.......X.....X........X.X..X.] duration:
43.091797452s trace:
https://jaeger.dev.storj.io/trace/6f01b97f3ddb54a7
FAIL (2026-05-16 00:05:29 UTC)
orbiter warning: context deadline exceeded
Duration: 43.091812471s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.....X.......X..............................X...X...........] duration:
4.297190407s trace:
https://jaeger.dev.storj.io/trace/34b4425d7c2a9299
FAIL (2026-05-16 00:05:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.297214374s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [......X.......X..............................X...X..........] duration:
4.055076646s trace:
https://jaeger.dev.storj.io/trace/34b4425d7c2a9299
FAIL (2026-05-16 00:04:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.055120669s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [..X.X..X.X..XX.X.X..XXX......X.X.......X.....X........X.X..X] duration:
1m0.001507192s trace:
https://jaeger.dev.storj.io/trace/6f01b97f3ddb54a7
FAIL (2026-05-16 00:04:46 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.001515468s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
15.395249424s trace:
https://jaeger.dev.storj.io/trace/7aee1db2369a7212
FAIL (2026-05-16 00:04:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
May 15 23:31:03.793482679 (364h43m14.314306424s) [upload=349446]: Starting upload
May 15 23:31:03.793539768 (364h43m14.314363513s) [upload=349446]: Done starting upload
May 15 23:31:03.793632195 (364h43m14.31445595s) [upload=349446]: Uploading segments...
May 15 23:31:03.793727968 (364h43m14.314551713s) [upload=349446]: Got next segment. Inline: false
May 15 23:31:03.793759228 (364h43m14.314582973s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
May 15 23:31:03.793763295 (364h43m14.31458704s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
May 15 23:31:04.028983593 (364h43m14.549807338s) [seg_pos={0 0}, upload=349446]: Begin upload segment...
May 15 23:31:04.029042185 (364h43m14.54986593s) [seg_pos={0 0}, upload=349446]: Done begin upload segment.
May 15 23:31:04.029098292 (364h43m14.549922037s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029123639 (364h43m14.549947384s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029560293 (364h43m14.550384038s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.029922404 (364h43m14.550746149s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.030246533 (364h43m14.551070278s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031211294 (364h43m14.552035039s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031574617 (364h43m14.552398362s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.031898575 (364h43m14.55272233s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.032633186 (364h43m14.553456932s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.032940222 (364h43m14.553763967s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.034094576 (364h43m14.554918321s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.035315475 (364h43m14.55613922s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.036723662 (364h43m14.557547407s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.038290051 (364h43m14.559113806s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.039157376 (364h43m14.559981121s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:04.040322049 (364h43m14.561145794s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Uploading piece...
May 15 23:31:05.861890230 (364h43m16.382713975s) [upload=349446]: Next returned nil segment
May 15 23:31:05.861903445 (364h43m16.382727191s) [upload=349446]: Waiting for error group managing segments...
May 15 23:31:06.164064150 (364h43m16.684887895s) [noise=true , address=145.79.227.44:4195 , piece_id=NR5QMITL5XVJTURIKKB4PMDARJ44L7KPSZ7DPD3CJLGOGWEVRWZA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.186906637 (364h43m16.707730382s) [noise=true , address=145.79.227.45:4234 , piece_id=7OAP23E2BKHVC3FR752ZYFKFAKSXQAIKOCDXF5JASZN2P4YG4Z4A, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.194445543 (364h43m16.715269298s) [noise=true , address=145.79.227.46:4604 , piece_id=CRJRL3YSIAHYBL6QINMAKHTJ7CGOW5DQGLNCSVIA5K2ZASNBNBIQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.199312090 (364h43m16.720135845s) [noise=true , address=145.79.227.49:4342 , piece_id=UK6XELYV24OZE655YBPTQ2CH64WX744TJO7RCXWNSJLZFDZNR6BQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.244272974 (364h43m16.765096719s) [noise=true , address=145.79.227.30:4761 , piece_id=EECP4RKW55BOO7XKUX7OOYYRELOAQ5Y3XDIFCIDJW74UJAG7DHQQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.264323375 (364h43m16.78514712s) [noise=true , address=145.79.227.13:4805 , piece_id=7GDWIAX2U4T5B2MXKMC4OQXTAAITT374YOOWZJKDID5QVWKZDF5Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.266214572 (364h43m16.787038318s) [noise=true , address=145.79.227.35:4485 , piece_id=EXJY6S6I3UXXWI2Q7LWDTBT5SY53RACYQPUSPHSKICRLGYSWHTSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.273598813 (364h43m16.794422559s) [noise=true , address=145.79.227.38:4979 , piece_id=WBZXPALQIDW5G66N6O53YGETI4BRKAUTRMD4INKMEK7J3KTEVF2Q, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.370539316 (364h43m16.891363061s) [noise=true , address=145.79.227.51:4411 , piece_id=AGVVI6FN5662XLAHHA3RECIO2GJBFPMMH6RH3IUEXWOGXMUITTHQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.607676541 (364h43m17.128500286s) [noise=true , address=45.91.116.14:4216 , piece_id=BPK5UQOU2C5II2Z4T3RTEDJEZCWIIHWK7S65KHOFUWZC7JYI4MUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.654258960 (364h43m17.175082705s) [noise=true , address=45.91.116.13:4155 , piece_id=UDIMRBB5FFNOJX7LQTUFGXZJHCB74VESND34EOSUBTF2LRUEONSA, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.716530637 (364h43m17.237354382s) [noise=true , address=45.91.116.12:4228 , piece_id=B5F2JRHPHOEAD3OK55HOLIBP3CZSIP6EWS65FVWR7VDWMZAJMQEQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.765454765 (364h43m17.28627851s) [noise=true , address=45.91.116.8:4045 , piece_id=HZ4HLGEQPOKUIYTWFNB4CI6PJRFLBQCWRGLZUCSHRCYLCBWC5URQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: <nil>
May 15 23:31:06.765475484 (364h43m17.286299229s) [seg_pos={0 0}, upload=349446]: Satisfied cohort requirements after 13 pieces
May 15 23:31:06.765754036 (364h43m17.286577781s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765760748 (364h43m17.286584493s) [noise=true , address=145.79.227.47:4267 , piece_id=VIRHSC72GSN6564PIA5ANACC2N2JPMS6VN3DHTPPZVD4S2K5TQUQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765770097 (364h43m17.286593842s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765807308 (364h43m17.286631053s) [noise=true , address=145.79.227.39:4014 , piece_id=4AA6E7XVVOKB6VO7WDDA7ZK533ZYPQ4WIOCPRBH5D36URDMADBWQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765921936 (364h43m17.286745682s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
May 15 23:31:06.765928649 (364h43m17.286752394s) [noise=true , address=45.91.116.5:4300 , piece_id=ARGGEEHMGE6V7JZD5RZF5TSGBEVURLMLXZP5MIMSHA6LFMMDKKPQ, seg_pos={0 0}, upload=349446]: Long tail cancellation due to optimal threshold
May 15 23:31:06.765981260 (364h43m17.286804995s) [seg_pos={0 0}, upload=349446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
May 15 23:31:06.766059889 (364h43m17.286883634s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 15 23:31:06.766065019 (364h43m17.286888764s) [upload=349446]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 15 23:31:07.102341032 (364h43m17.623164777s) [upload=349446]: Done uploading segments...
May 15 23:31:07.102359026 (364h43m17.623182771s) [upload=349446]: Upload finished. err: <nil>
Duration: 15.395270795s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.......X.......X..............................X...X.........] duration:
4.16333845s trace:
https://jaeger.dev.storj.io/trace/34b4425d7c2a9299
FAIL (2026-05-16 00:03:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.163361843s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
17 out of 60 is failed (threshold=3) [...X.X..X.X..XX.X.X..XXX......X.X.......X.....X........X.X..] duration:
47.572959847s trace:
https://jaeger.dev.storj.io/trace/1b31ff88ac2c14f2
FAIL (2026-05-16 00:03:46 UTC)
orbiter warning: context deadline exceeded
Duration: 47.572968724s
core
uplink
us-select-1
us1-placement-20
|