Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
10 out of 60 is failed (threshold=3) [..............X.......X.XXX...X........X..X.............XX..] duration:
6.48203948s trace:
https://jaeger.dev.storj.io/trace/3f402cdaf22e5c
FAIL (2026-05-26 01:02:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.482085427s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
10 out of 60 is failed (threshold=3) [...............X.......X.XXX...X........X..X.............XX.] duration:
31.584271405s trace:
https://jaeger.dev.storj.io/trace/3f402cdaf22e5c
FAIL (2026-05-26 01:02:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 31.584280712s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
10 out of 60 is failed (threshold=3) [................X.......X.XXX...X........X..X.............XX] duration:
1m0.000946552s trace:
https://jaeger.dev.storj.io/trace/3f402cdaf22e5c
FAIL (2026-05-26 01:02:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000955299s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
9 out of 60 is failed (threshold=3) [.................X.......X.XXX...X........X..X.............X] duration:
1m0.001273966s trace:
https://jaeger.dev.storj.io/trace/1bdd134e80798125
FAIL (2026-05-26 01:01:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.00129281s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [..................X.......X.XXX...X........X..X.............] duration:
6.420095213s trace:
https://jaeger.dev.storj.io/trace/5b59e7787af4eb57
FAIL (2026-05-26 00:59:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.420115912s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [...................X.......X.XXX...X........X..X............] duration:
6.720450189s trace:
https://jaeger.dev.storj.io/trace/5b59e7787af4eb57
FAIL (2026-05-26 00:58:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.720518851s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
9 out of 60 is failed (threshold=3) [X...................X.......X.XXX...X........X..X...........] duration:
6.705344841s trace:
https://jaeger.dev.storj.io/trace/5b59e7787af4eb57
FAIL (2026-05-26 00:57:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.705366543s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
4.207263583s trace:
https://jaeger.dev.storj.io/trace/54dd3f55a77d17ef
FAIL (2026-05-26 00:56:53 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320420647 (22h6m37.103760746s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320702816 (22h6m37.104042915s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320993440 (22h6m37.104333539s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321256863 (22h6m37.104596962s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321827863 (22h6m37.105167962s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321916452 (22h6m37.105256551s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322280286 (22h6m37.105620385s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322452585 (22h6m37.105792684s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322950865 (22h6m37.106290964s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323536944 (22h6m37.106877043s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323731265 (22h6m37.107071364s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324123954 (22h6m37.107464053s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324414138 (22h6m37.107754227s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.325132458 (22h6m37.108472547s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.327055307 (22h6m37.110395406s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:52.388936548 (22h6m38.172276647s) [upload=21163]: Next returned nil segment
May 26 00:10:52.388980332 (22h6m38.172320431s) [upload=21163]: Waiting for error group managing segments...
May 26 00:10:52.674703801 (22h6m38.4580439s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.676576304 (22h6m38.459916403s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.685919140 (22h6m38.469259239s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.691062223 (22h6m38.474402322s) [noise=true , address=212.102.52.214:4347 , piece_id=OP75SIPBPG6SRO427YW5DXBI2LX3IGJYQ272EQABJCYXPJQSX3IA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.692733643 (22h6m38.476073742s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.695375413 (22h6m38.478715512s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.697960065 (22h6m38.481300164s) [noise=true , address=212.102.52.215:4390 , piece_id=E4XO5YC7J2INYSCZLMJO6VRV4DMDM4YDBYAOIHXR7NHCC5EZELKQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.699281385 (22h6m38.482621484s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.701163387 (22h6m38.484503486s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.702857559 (22h6m38.486197658s) [noise=true , address=212.102.52.214:4353 , piece_id=WZG7PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.705263059 (22h6m38.488603159s) [noise=true , address=212.102.52.215:4386 , piece_id=S5YIYIWUEMKONHJACR7E7PO7U7N2FMU3GH3OO4ULB77LLLC624MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707419504 (22h6m38.490759593s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707786013 (22h6m38.491126112s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.709872454 (22h6m38.493212543s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.730564756 (22h6m38.513904845s) [noise=true , address=212.102.52.215:4382 , piece_id=BAJ5RC7425LFHND3DAXX4YSESMOUIRZOH2SHWHITX4KT3OAE54YQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736735980 (22h6m38.520076079s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736752663 (22h6m38.520092762s) [seg_pos={0 0}, upload=21163]: Satisfied cohort requirements after 16 pieces
May 26 00:10:52.736921284 (22h6m38.520261383s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f4fqtM26XZspG77Sjho67Tx9ucw1hAwg98xbPxqUcE6f1nH5U): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.736929240 (22h6m38.520269339s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737040080 (22h6m38.520380179s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SUcotB1144nMnt3JXXbVyNoz9aS6352uH9cGptt4RqKbvQkRv): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737054549 (22h6m38.520394678s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737102950 (22h6m38.520443049s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1272rh1tzNBuEmXqx4MpBe7NKXPac2wuNPCEpYqFk56gTrodcU2): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737108672 (22h6m38.520448771s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737196889 (22h6m38.520536988s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f1unaJrBkG1zw5XdBmCMeS8EKdqELYbQcr5cLq513xvctxcED): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737201459 (22h6m38.520541558s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737216567 (22h6m38.520556666s) [seg_pos={0 0}, upload=21163]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 00:10:52.737295508 (22h6m38.520635597s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 00:10:52.737299646 (22h6m38.520639745s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 00:11:00.530433246 (22h6m46.313773355s) [upload=21163]: Done uploading segments...
May 26 00:11:00.530455227 (22h6m46.313795347s) [upload=21163]: Upload finished. err: <nil>
Duration: 4.207284873s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
9 out of 60 is failed (threshold=3) [.X...................X.......X.XXX...X........X..X..........] duration:
7.653247081s trace:
https://jaeger.dev.storj.io/trace/5b59e7787af4eb57
FAIL (2026-05-26 00:56:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.653265798s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
4.034266888s trace:
https://jaeger.dev.storj.io/trace/54dd3f55a77d17ef
FAIL (2026-05-26 00:55:52 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320420647 (22h6m37.103760746s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320702816 (22h6m37.104042915s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320993440 (22h6m37.104333539s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321256863 (22h6m37.104596962s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321827863 (22h6m37.105167962s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321916452 (22h6m37.105256551s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322280286 (22h6m37.105620385s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322452585 (22h6m37.105792684s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322950865 (22h6m37.106290964s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323536944 (22h6m37.106877043s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323731265 (22h6m37.107071364s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324123954 (22h6m37.107464053s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324414138 (22h6m37.107754227s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.325132458 (22h6m37.108472547s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.327055307 (22h6m37.110395406s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:52.388936548 (22h6m38.172276647s) [upload=21163]: Next returned nil segment
May 26 00:10:52.388980332 (22h6m38.172320431s) [upload=21163]: Waiting for error group managing segments...
May 26 00:10:52.674703801 (22h6m38.4580439s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.676576304 (22h6m38.459916403s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.685919140 (22h6m38.469259239s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.691062223 (22h6m38.474402322s) [noise=true , address=212.102.52.214:4347 , piece_id=OP75SIPBPG6SRO427YW5DXBI2LX3IGJYQ272EQABJCYXPJQSX3IA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.692733643 (22h6m38.476073742s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.695375413 (22h6m38.478715512s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.697960065 (22h6m38.481300164s) [noise=true , address=212.102.52.215:4390 , piece_id=E4XO5YC7J2INYSCZLMJO6VRV4DMDM4YDBYAOIHXR7NHCC5EZELKQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.699281385 (22h6m38.482621484s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.701163387 (22h6m38.484503486s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.702857559 (22h6m38.486197658s) [noise=true , address=212.102.52.214:4353 , piece_id=WZG7PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.705263059 (22h6m38.488603159s) [noise=true , address=212.102.52.215:4386 , piece_id=S5YIYIWUEMKONHJACR7E7PO7U7N2FMU3GH3OO4ULB77LLLC624MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707419504 (22h6m38.490759593s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707786013 (22h6m38.491126112s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.709872454 (22h6m38.493212543s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.730564756 (22h6m38.513904845s) [noise=true , address=212.102.52.215:4382 , piece_id=BAJ5RC7425LFHND3DAXX4YSESMOUIRZOH2SHWHITX4KT3OAE54YQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736735980 (22h6m38.520076079s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736752663 (22h6m38.520092762s) [seg_pos={0 0}, upload=21163]: Satisfied cohort requirements after 16 pieces
May 26 00:10:52.736921284 (22h6m38.520261383s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f4fqtM26XZspG77Sjho67Tx9ucw1hAwg98xbPxqUcE6f1nH5U): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.736929240 (22h6m38.520269339s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737040080 (22h6m38.520380179s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SUcotB1144nMnt3JXXbVyNoz9aS6352uH9cGptt4RqKbvQkRv): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737054549 (22h6m38.520394678s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737102950 (22h6m38.520443049s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1272rh1tzNBuEmXqx4MpBe7NKXPac2wuNPCEpYqFk56gTrodcU2): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737108672 (22h6m38.520448771s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737196889 (22h6m38.520536988s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f1unaJrBkG1zw5XdBmCMeS8EKdqELYbQcr5cLq513xvctxcED): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737201459 (22h6m38.520541558s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737216567 (22h6m38.520556666s) [seg_pos={0 0}, upload=21163]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 00:10:52.737295508 (22h6m38.520635597s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 00:10:52.737299646 (22h6m38.520639745s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 00:11:00.530433246 (22h6m46.313773355s) [upload=21163]: Done uploading segments...
May 26 00:11:00.530455227 (22h6m46.313795347s) [upload=21163]: Upload finished. err: <nil>
Duration: 4.034285042s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
9 out of 60 is failed (threshold=3) [..X...................X.......X.XXX...X........X..X.........] duration:
6.7272778s trace:
https://jaeger.dev.storj.io/trace/5b59e7787af4eb57
FAIL (2026-05-26 00:55:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.727298259s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
9 out of 60 is failed (threshold=3) [...X...................X.......X.XXX...X........X..X........] duration:
6.432809995s trace:
https://jaeger.dev.storj.io/trace/5b59e7787af4eb57
FAIL (2026-05-26 00:55:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.432835885s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
4.023495709s trace:
https://jaeger.dev.storj.io/trace/54dd3f55a77d17ef
FAIL (2026-05-26 00:55:13 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320420647 (22h6m37.103760746s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320702816 (22h6m37.104042915s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320993440 (22h6m37.104333539s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321256863 (22h6m37.104596962s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321827863 (22h6m37.105167962s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321916452 (22h6m37.105256551s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322280286 (22h6m37.105620385s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322452585 (22h6m37.105792684s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322950865 (22h6m37.106290964s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323536944 (22h6m37.106877043s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323731265 (22h6m37.107071364s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324123954 (22h6m37.107464053s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324414138 (22h6m37.107754227s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.325132458 (22h6m37.108472547s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.327055307 (22h6m37.110395406s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:52.388936548 (22h6m38.172276647s) [upload=21163]: Next returned nil segment
May 26 00:10:52.388980332 (22h6m38.172320431s) [upload=21163]: Waiting for error group managing segments...
May 26 00:10:52.674703801 (22h6m38.4580439s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.676576304 (22h6m38.459916403s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.685919140 (22h6m38.469259239s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.691062223 (22h6m38.474402322s) [noise=true , address=212.102.52.214:4347 , piece_id=OP75SIPBPG6SRO427YW5DXBI2LX3IGJYQ272EQABJCYXPJQSX3IA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.692733643 (22h6m38.476073742s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.695375413 (22h6m38.478715512s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.697960065 (22h6m38.481300164s) [noise=true , address=212.102.52.215:4390 , piece_id=E4XO5YC7J2INYSCZLMJO6VRV4DMDM4YDBYAOIHXR7NHCC5EZELKQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.699281385 (22h6m38.482621484s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.701163387 (22h6m38.484503486s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.702857559 (22h6m38.486197658s) [noise=true , address=212.102.52.214:4353 , piece_id=WZG7PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.705263059 (22h6m38.488603159s) [noise=true , address=212.102.52.215:4386 , piece_id=S5YIYIWUEMKONHJACR7E7PO7U7N2FMU3GH3OO4ULB77LLLC624MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707419504 (22h6m38.490759593s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707786013 (22h6m38.491126112s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.709872454 (22h6m38.493212543s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.730564756 (22h6m38.513904845s) [noise=true , address=212.102.52.215:4382 , piece_id=BAJ5RC7425LFHND3DAXX4YSESMOUIRZOH2SHWHITX4KT3OAE54YQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736735980 (22h6m38.520076079s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736752663 (22h6m38.520092762s) [seg_pos={0 0}, upload=21163]: Satisfied cohort requirements after 16 pieces
May 26 00:10:52.736921284 (22h6m38.520261383s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f4fqtM26XZspG77Sjho67Tx9ucw1hAwg98xbPxqUcE6f1nH5U): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.736929240 (22h6m38.520269339s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737040080 (22h6m38.520380179s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SUcotB1144nMnt3JXXbVyNoz9aS6352uH9cGptt4RqKbvQkRv): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737054549 (22h6m38.520394678s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737102950 (22h6m38.520443049s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1272rh1tzNBuEmXqx4MpBe7NKXPac2wuNPCEpYqFk56gTrodcU2): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737108672 (22h6m38.520448771s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737196889 (22h6m38.520536988s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f1unaJrBkG1zw5XdBmCMeS8EKdqELYbQcr5cLq513xvctxcED): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737201459 (22h6m38.520541558s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737216567 (22h6m38.520556666s) [seg_pos={0 0}, upload=21163]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 00:10:52.737295508 (22h6m38.520635597s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 00:10:52.737299646 (22h6m38.520639745s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 00:11:00.530433246 (22h6m46.313773355s) [upload=21163]: Done uploading segments...
May 26 00:11:00.530455227 (22h6m46.313795347s) [upload=21163]: Upload finished. err: <nil>
Duration: 4.023518943s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
9 out of 60 is failed (threshold=3) [....X...................X.......X.XXX...X........X..X.......] duration:
6.581172827s trace:
https://jaeger.dev.storj.io/trace/5b59e7787af4eb57
FAIL (2026-05-26 00:54:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.581230106s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [X..X........................................................] duration:
7.988389931s trace:
https://jaeger.dev.storj.io/trace/54dd3f55a77d17ef
FAIL (2026-05-26 00:54:17 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320420647 (22h6m37.103760746s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320702816 (22h6m37.104042915s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320993440 (22h6m37.104333539s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321256863 (22h6m37.104596962s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321827863 (22h6m37.105167962s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321916452 (22h6m37.105256551s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322280286 (22h6m37.105620385s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322452585 (22h6m37.105792684s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322950865 (22h6m37.106290964s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323536944 (22h6m37.106877043s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323731265 (22h6m37.107071364s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324123954 (22h6m37.107464053s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324414138 (22h6m37.107754227s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.325132458 (22h6m37.108472547s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.327055307 (22h6m37.110395406s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:52.388936548 (22h6m38.172276647s) [upload=21163]: Next returned nil segment
May 26 00:10:52.388980332 (22h6m38.172320431s) [upload=21163]: Waiting for error group managing segments...
May 26 00:10:52.674703801 (22h6m38.4580439s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.676576304 (22h6m38.459916403s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.685919140 (22h6m38.469259239s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.691062223 (22h6m38.474402322s) [noise=true , address=212.102.52.214:4347 , piece_id=OP75SIPBPG6SRO427YW5DXBI2LX3IGJYQ272EQABJCYXPJQSX3IA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.692733643 (22h6m38.476073742s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.695375413 (22h6m38.478715512s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.697960065 (22h6m38.481300164s) [noise=true , address=212.102.52.215:4390 , piece_id=E4XO5YC7J2INYSCZLMJO6VRV4DMDM4YDBYAOIHXR7NHCC5EZELKQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.699281385 (22h6m38.482621484s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.701163387 (22h6m38.484503486s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.702857559 (22h6m38.486197658s) [noise=true , address=212.102.52.214:4353 , piece_id=WZG7PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.705263059 (22h6m38.488603159s) [noise=true , address=212.102.52.215:4386 , piece_id=S5YIYIWUEMKONHJACR7E7PO7U7N2FMU3GH3OO4ULB77LLLC624MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707419504 (22h6m38.490759593s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707786013 (22h6m38.491126112s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.709872454 (22h6m38.493212543s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.730564756 (22h6m38.513904845s) [noise=true , address=212.102.52.215:4382 , piece_id=BAJ5RC7425LFHND3DAXX4YSESMOUIRZOH2SHWHITX4KT3OAE54YQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736735980 (22h6m38.520076079s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736752663 (22h6m38.520092762s) [seg_pos={0 0}, upload=21163]: Satisfied cohort requirements after 16 pieces
May 26 00:10:52.736921284 (22h6m38.520261383s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f4fqtM26XZspG77Sjho67Tx9ucw1hAwg98xbPxqUcE6f1nH5U): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.736929240 (22h6m38.520269339s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737040080 (22h6m38.520380179s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SUcotB1144nMnt3JXXbVyNoz9aS6352uH9cGptt4RqKbvQkRv): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737054549 (22h6m38.520394678s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737102950 (22h6m38.520443049s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1272rh1tzNBuEmXqx4MpBe7NKXPac2wuNPCEpYqFk56gTrodcU2): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737108672 (22h6m38.520448771s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737196889 (22h6m38.520536988s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f1unaJrBkG1zw5XdBmCMeS8EKdqELYbQcr5cLq513xvctxcED): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737201459 (22h6m38.520541558s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737216567 (22h6m38.520556666s) [seg_pos={0 0}, upload=21163]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 00:10:52.737295508 (22h6m38.520635597s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 00:10:52.737299646 (22h6m38.520639745s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 00:11:00.530433246 (22h6m46.313773355s) [upload=21163]: Done uploading segments...
May 26 00:11:00.530455227 (22h6m46.313795347s) [upload=21163]: Upload finished. err: <nil>
Duration: 7.988416991s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
9 out of 60 is failed (threshold=3) [.....X...................X.......X.XXX...X........X..X......] duration:
23.225508262s trace:
https://jaeger.dev.storj.io/trace/5b59e7787af4eb57
FAIL (2026-05-26 00:54:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 23.225525124s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [.X..X.......................................................] duration:
4.076656139s trace:
https://jaeger.dev.storj.io/trace/54dd3f55a77d17ef
FAIL (2026-05-26 00:53:49 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320420647 (22h6m37.103760746s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320702816 (22h6m37.104042915s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320993440 (22h6m37.104333539s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321256863 (22h6m37.104596962s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321827863 (22h6m37.105167962s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321916452 (22h6m37.105256551s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322280286 (22h6m37.105620385s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322452585 (22h6m37.105792684s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322950865 (22h6m37.106290964s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323536944 (22h6m37.106877043s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323731265 (22h6m37.107071364s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324123954 (22h6m37.107464053s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324414138 (22h6m37.107754227s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.325132458 (22h6m37.108472547s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.327055307 (22h6m37.110395406s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:52.388936548 (22h6m38.172276647s) [upload=21163]: Next returned nil segment
May 26 00:10:52.388980332 (22h6m38.172320431s) [upload=21163]: Waiting for error group managing segments...
May 26 00:10:52.674703801 (22h6m38.4580439s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.676576304 (22h6m38.459916403s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.685919140 (22h6m38.469259239s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.691062223 (22h6m38.474402322s) [noise=true , address=212.102.52.214:4347 , piece_id=OP75SIPBPG6SRO427YW5DXBI2LX3IGJYQ272EQABJCYXPJQSX3IA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.692733643 (22h6m38.476073742s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.695375413 (22h6m38.478715512s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.697960065 (22h6m38.481300164s) [noise=true , address=212.102.52.215:4390 , piece_id=E4XO5YC7J2INYSCZLMJO6VRV4DMDM4YDBYAOIHXR7NHCC5EZELKQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.699281385 (22h6m38.482621484s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.701163387 (22h6m38.484503486s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.702857559 (22h6m38.486197658s) [noise=true , address=212.102.52.214:4353 , piece_id=WZG7PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.705263059 (22h6m38.488603159s) [noise=true , address=212.102.52.215:4386 , piece_id=S5YIYIWUEMKONHJACR7E7PO7U7N2FMU3GH3OO4ULB77LLLC624MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707419504 (22h6m38.490759593s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707786013 (22h6m38.491126112s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.709872454 (22h6m38.493212543s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.730564756 (22h6m38.513904845s) [noise=true , address=212.102.52.215:4382 , piece_id=BAJ5RC7425LFHND3DAXX4YSESMOUIRZOH2SHWHITX4KT3OAE54YQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736735980 (22h6m38.520076079s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736752663 (22h6m38.520092762s) [seg_pos={0 0}, upload=21163]: Satisfied cohort requirements after 16 pieces
May 26 00:10:52.736921284 (22h6m38.520261383s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f4fqtM26XZspG77Sjho67Tx9ucw1hAwg98xbPxqUcE6f1nH5U): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.736929240 (22h6m38.520269339s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737040080 (22h6m38.520380179s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SUcotB1144nMnt3JXXbVyNoz9aS6352uH9cGptt4RqKbvQkRv): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737054549 (22h6m38.520394678s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737102950 (22h6m38.520443049s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1272rh1tzNBuEmXqx4MpBe7NKXPac2wuNPCEpYqFk56gTrodcU2): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737108672 (22h6m38.520448771s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737196889 (22h6m38.520536988s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f1unaJrBkG1zw5XdBmCMeS8EKdqELYbQcr5cLq513xvctxcED): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737201459 (22h6m38.520541558s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737216567 (22h6m38.520556666s) [seg_pos={0 0}, upload=21163]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 00:10:52.737295508 (22h6m38.520635597s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 00:10:52.737299646 (22h6m38.520639745s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 00:11:00.530433246 (22h6m46.313773355s) [upload=21163]: Done uploading segments...
May 26 00:11:00.530455227 (22h6m46.313795347s) [upload=21163]: Upload finished. err: <nil>
Duration: 4.076676988s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [..X..X......................................................] duration:
4.807341466s trace:
https://jaeger.dev.storj.io/trace/54dd3f55a77d17ef
FAIL (2026-05-26 00:53:20 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320420647 (22h6m37.103760746s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320702816 (22h6m37.104042915s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320993440 (22h6m37.104333539s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321256863 (22h6m37.104596962s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321827863 (22h6m37.105167962s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321916452 (22h6m37.105256551s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322280286 (22h6m37.105620385s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322452585 (22h6m37.105792684s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322950865 (22h6m37.106290964s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323536944 (22h6m37.106877043s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323731265 (22h6m37.107071364s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324123954 (22h6m37.107464053s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324414138 (22h6m37.107754227s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.325132458 (22h6m37.108472547s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.327055307 (22h6m37.110395406s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:52.388936548 (22h6m38.172276647s) [upload=21163]: Next returned nil segment
May 26 00:10:52.388980332 (22h6m38.172320431s) [upload=21163]: Waiting for error group managing segments...
May 26 00:10:52.674703801 (22h6m38.4580439s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.676576304 (22h6m38.459916403s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.685919140 (22h6m38.469259239s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.691062223 (22h6m38.474402322s) [noise=true , address=212.102.52.214:4347 , piece_id=OP75SIPBPG6SRO427YW5DXBI2LX3IGJYQ272EQABJCYXPJQSX3IA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.692733643 (22h6m38.476073742s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.695375413 (22h6m38.478715512s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.697960065 (22h6m38.481300164s) [noise=true , address=212.102.52.215:4390 , piece_id=E4XO5YC7J2INYSCZLMJO6VRV4DMDM4YDBYAOIHXR7NHCC5EZELKQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.699281385 (22h6m38.482621484s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.701163387 (22h6m38.484503486s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.702857559 (22h6m38.486197658s) [noise=true , address=212.102.52.214:4353 , piece_id=WZG7PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.705263059 (22h6m38.488603159s) [noise=true , address=212.102.52.215:4386 , piece_id=S5YIYIWUEMKONHJACR7E7PO7U7N2FMU3GH3OO4ULB77LLLC624MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707419504 (22h6m38.490759593s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707786013 (22h6m38.491126112s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.709872454 (22h6m38.493212543s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.730564756 (22h6m38.513904845s) [noise=true , address=212.102.52.215:4382 , piece_id=BAJ5RC7425LFHND3DAXX4YSESMOUIRZOH2SHWHITX4KT3OAE54YQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736735980 (22h6m38.520076079s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736752663 (22h6m38.520092762s) [seg_pos={0 0}, upload=21163]: Satisfied cohort requirements after 16 pieces
May 26 00:10:52.736921284 (22h6m38.520261383s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f4fqtM26XZspG77Sjho67Tx9ucw1hAwg98xbPxqUcE6f1nH5U): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.736929240 (22h6m38.520269339s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737040080 (22h6m38.520380179s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SUcotB1144nMnt3JXXbVyNoz9aS6352uH9cGptt4RqKbvQkRv): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737054549 (22h6m38.520394678s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737102950 (22h6m38.520443049s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1272rh1tzNBuEmXqx4MpBe7NKXPac2wuNPCEpYqFk56gTrodcU2): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737108672 (22h6m38.520448771s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737196889 (22h6m38.520536988s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f1unaJrBkG1zw5XdBmCMeS8EKdqELYbQcr5cLq513xvctxcED): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737201459 (22h6m38.520541558s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737216567 (22h6m38.520556666s) [seg_pos={0 0}, upload=21163]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 00:10:52.737295508 (22h6m38.520635597s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 00:10:52.737299646 (22h6m38.520639745s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 00:11:00.530433246 (22h6m46.313773355s) [upload=21163]: Done uploading segments...
May 26 00:11:00.530455227 (22h6m46.313795347s) [upload=21163]: Upload finished. err: <nil>
Duration: 4.807366364s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
9 out of 60 is failed (threshold=3) [......X...................X.......X.XXX...X........X..X.....] duration:
15.567787472s trace:
https://jaeger.dev.storj.io/trace/5b59e7787af4eb57
FAIL (2026-05-26 00:52:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 15.567808844s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [...X..X.....................................................] duration:
4.273168293s trace:
https://jaeger.dev.storj.io/trace/54dd3f55a77d17ef
FAIL (2026-05-26 00:52:48 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320420647 (22h6m37.103760746s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320702816 (22h6m37.104042915s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320993440 (22h6m37.104333539s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321256863 (22h6m37.104596962s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321827863 (22h6m37.105167962s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321916452 (22h6m37.105256551s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322280286 (22h6m37.105620385s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322452585 (22h6m37.105792684s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322950865 (22h6m37.106290964s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323536944 (22h6m37.106877043s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323731265 (22h6m37.107071364s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324123954 (22h6m37.107464053s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324414138 (22h6m37.107754227s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.325132458 (22h6m37.108472547s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.327055307 (22h6m37.110395406s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:52.388936548 (22h6m38.172276647s) [upload=21163]: Next returned nil segment
May 26 00:10:52.388980332 (22h6m38.172320431s) [upload=21163]: Waiting for error group managing segments...
May 26 00:10:52.674703801 (22h6m38.4580439s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.676576304 (22h6m38.459916403s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.685919140 (22h6m38.469259239s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.691062223 (22h6m38.474402322s) [noise=true , address=212.102.52.214:4347 , piece_id=OP75SIPBPG6SRO427YW5DXBI2LX3IGJYQ272EQABJCYXPJQSX3IA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.692733643 (22h6m38.476073742s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.695375413 (22h6m38.478715512s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.697960065 (22h6m38.481300164s) [noise=true , address=212.102.52.215:4390 , piece_id=E4XO5YC7J2INYSCZLMJO6VRV4DMDM4YDBYAOIHXR7NHCC5EZELKQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.699281385 (22h6m38.482621484s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.701163387 (22h6m38.484503486s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.702857559 (22h6m38.486197658s) [noise=true , address=212.102.52.214:4353 , piece_id=WZG7PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.705263059 (22h6m38.488603159s) [noise=true , address=212.102.52.215:4386 , piece_id=S5YIYIWUEMKONHJACR7E7PO7U7N2FMU3GH3OO4ULB77LLLC624MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707419504 (22h6m38.490759593s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707786013 (22h6m38.491126112s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.709872454 (22h6m38.493212543s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.730564756 (22h6m38.513904845s) [noise=true , address=212.102.52.215:4382 , piece_id=BAJ5RC7425LFHND3DAXX4YSESMOUIRZOH2SHWHITX4KT3OAE54YQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736735980 (22h6m38.520076079s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736752663 (22h6m38.520092762s) [seg_pos={0 0}, upload=21163]: Satisfied cohort requirements after 16 pieces
May 26 00:10:52.736921284 (22h6m38.520261383s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f4fqtM26XZspG77Sjho67Tx9ucw1hAwg98xbPxqUcE6f1nH5U): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.736929240 (22h6m38.520269339s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737040080 (22h6m38.520380179s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SUcotB1144nMnt3JXXbVyNoz9aS6352uH9cGptt4RqKbvQkRv): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737054549 (22h6m38.520394678s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737102950 (22h6m38.520443049s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1272rh1tzNBuEmXqx4MpBe7NKXPac2wuNPCEpYqFk56gTrodcU2): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737108672 (22h6m38.520448771s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737196889 (22h6m38.520536988s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f1unaJrBkG1zw5XdBmCMeS8EKdqELYbQcr5cLq513xvctxcED): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737201459 (22h6m38.520541558s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737216567 (22h6m38.520556666s) [seg_pos={0 0}, upload=21163]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 00:10:52.737295508 (22h6m38.520635597s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 00:10:52.737299646 (22h6m38.520639745s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 00:11:00.530433246 (22h6m46.313773355s) [upload=21163]: Done uploading segments...
May 26 00:11:00.530455227 (22h6m46.313795347s) [upload=21163]: Upload finished. err: <nil>
Duration: 4.273193992s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
9 out of 60 is failed (threshold=3) [.......X...................X.......X.XXX...X........X..X....] duration:
6.66378779s trace:
https://jaeger.dev.storj.io/trace/5b59e7787af4eb57
FAIL (2026-05-26 00:52:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.663813269s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [....X..X....................................................] duration:
6.448549979s trace:
https://jaeger.dev.storj.io/trace/54dd3f55a77d17ef
FAIL (2026-05-26 00:51:38 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320420647 (22h6m37.103760746s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320702816 (22h6m37.104042915s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320993440 (22h6m37.104333539s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321256863 (22h6m37.104596962s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321827863 (22h6m37.105167962s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321916452 (22h6m37.105256551s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322280286 (22h6m37.105620385s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322452585 (22h6m37.105792684s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322950865 (22h6m37.106290964s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323536944 (22h6m37.106877043s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323731265 (22h6m37.107071364s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324123954 (22h6m37.107464053s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324414138 (22h6m37.107754227s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.325132458 (22h6m37.108472547s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.327055307 (22h6m37.110395406s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:52.388936548 (22h6m38.172276647s) [upload=21163]: Next returned nil segment
May 26 00:10:52.388980332 (22h6m38.172320431s) [upload=21163]: Waiting for error group managing segments...
May 26 00:10:52.674703801 (22h6m38.4580439s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.676576304 (22h6m38.459916403s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.685919140 (22h6m38.469259239s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.691062223 (22h6m38.474402322s) [noise=true , address=212.102.52.214:4347 , piece_id=OP75SIPBPG6SRO427YW5DXBI2LX3IGJYQ272EQABJCYXPJQSX3IA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.692733643 (22h6m38.476073742s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.695375413 (22h6m38.478715512s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.697960065 (22h6m38.481300164s) [noise=true , address=212.102.52.215:4390 , piece_id=E4XO5YC7J2INYSCZLMJO6VRV4DMDM4YDBYAOIHXR7NHCC5EZELKQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.699281385 (22h6m38.482621484s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.701163387 (22h6m38.484503486s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.702857559 (22h6m38.486197658s) [noise=true , address=212.102.52.214:4353 , piece_id=WZG7PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.705263059 (22h6m38.488603159s) [noise=true , address=212.102.52.215:4386 , piece_id=S5YIYIWUEMKONHJACR7E7PO7U7N2FMU3GH3OO4ULB77LLLC624MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707419504 (22h6m38.490759593s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707786013 (22h6m38.491126112s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.709872454 (22h6m38.493212543s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.730564756 (22h6m38.513904845s) [noise=true , address=212.102.52.215:4382 , piece_id=BAJ5RC7425LFHND3DAXX4YSESMOUIRZOH2SHWHITX4KT3OAE54YQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736735980 (22h6m38.520076079s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736752663 (22h6m38.520092762s) [seg_pos={0 0}, upload=21163]: Satisfied cohort requirements after 16 pieces
May 26 00:10:52.736921284 (22h6m38.520261383s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f4fqtM26XZspG77Sjho67Tx9ucw1hAwg98xbPxqUcE6f1nH5U): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.736929240 (22h6m38.520269339s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737040080 (22h6m38.520380179s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SUcotB1144nMnt3JXXbVyNoz9aS6352uH9cGptt4RqKbvQkRv): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737054549 (22h6m38.520394678s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737102950 (22h6m38.520443049s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1272rh1tzNBuEmXqx4MpBe7NKXPac2wuNPCEpYqFk56gTrodcU2): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737108672 (22h6m38.520448771s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737196889 (22h6m38.520536988s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f1unaJrBkG1zw5XdBmCMeS8EKdqELYbQcr5cLq513xvctxcED): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737201459 (22h6m38.520541558s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737216567 (22h6m38.520556666s) [seg_pos={0 0}, upload=21163]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 00:10:52.737295508 (22h6m38.520635597s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 00:10:52.737299646 (22h6m38.520639745s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 00:11:00.530433246 (22h6m46.313773355s) [upload=21163]: Done uploading segments...
May 26 00:11:00.530455227 (22h6m46.313795347s) [upload=21163]: Upload finished. err: <nil>
Duration: 6.448573173s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
9 out of 60 is failed (threshold=3) [........X...................X.......X.XXX...X........X..X...] duration:
7.728548062s trace:
https://jaeger.dev.storj.io/trace/5b59e7787af4eb57
FAIL (2026-05-26 00:51:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.72857269s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
9 out of 60 is failed (threshold=3) [.........X...................X.......X.XXX...X........X..X..] duration:
6.747320498s trace:
https://jaeger.dev.storj.io/trace/5b59e7787af4eb57
FAIL (2026-05-26 00:51:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.747335597s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
9 out of 60 is failed (threshold=3) [..........X...................X.......X.XXX...X........X..X.] duration:
30.469557466s trace:
https://jaeger.dev.storj.io/trace/5b59e7787af4eb57
FAIL (2026-05-26 00:50:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.469569139s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [.....X..X...................................................] duration:
3.908222131s trace:
https://jaeger.dev.storj.io/trace/54dd3f55a77d17ef
FAIL (2026-05-26 00:50:33 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320420647 (22h6m37.103760746s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320702816 (22h6m37.104042915s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320993440 (22h6m37.104333539s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321256863 (22h6m37.104596962s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321827863 (22h6m37.105167962s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321916452 (22h6m37.105256551s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322280286 (22h6m37.105620385s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322452585 (22h6m37.105792684s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322950865 (22h6m37.106290964s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323536944 (22h6m37.106877043s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323731265 (22h6m37.107071364s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324123954 (22h6m37.107464053s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324414138 (22h6m37.107754227s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.325132458 (22h6m37.108472547s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.327055307 (22h6m37.110395406s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:52.388936548 (22h6m38.172276647s) [upload=21163]: Next returned nil segment
May 26 00:10:52.388980332 (22h6m38.172320431s) [upload=21163]: Waiting for error group managing segments...
May 26 00:10:52.674703801 (22h6m38.4580439s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.676576304 (22h6m38.459916403s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.685919140 (22h6m38.469259239s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.691062223 (22h6m38.474402322s) [noise=true , address=212.102.52.214:4347 , piece_id=OP75SIPBPG6SRO427YW5DXBI2LX3IGJYQ272EQABJCYXPJQSX3IA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.692733643 (22h6m38.476073742s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.695375413 (22h6m38.478715512s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.697960065 (22h6m38.481300164s) [noise=true , address=212.102.52.215:4390 , piece_id=E4XO5YC7J2INYSCZLMJO6VRV4DMDM4YDBYAOIHXR7NHCC5EZELKQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.699281385 (22h6m38.482621484s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.701163387 (22h6m38.484503486s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.702857559 (22h6m38.486197658s) [noise=true , address=212.102.52.214:4353 , piece_id=WZG7PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.705263059 (22h6m38.488603159s) [noise=true , address=212.102.52.215:4386 , piece_id=S5YIYIWUEMKONHJACR7E7PO7U7N2FMU3GH3OO4ULB77LLLC624MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707419504 (22h6m38.490759593s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707786013 (22h6m38.491126112s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.709872454 (22h6m38.493212543s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.730564756 (22h6m38.513904845s) [noise=true , address=212.102.52.215:4382 , piece_id=BAJ5RC7425LFHND3DAXX4YSESMOUIRZOH2SHWHITX4KT3OAE54YQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736735980 (22h6m38.520076079s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736752663 (22h6m38.520092762s) [seg_pos={0 0}, upload=21163]: Satisfied cohort requirements after 16 pieces
May 26 00:10:52.736921284 (22h6m38.520261383s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f4fqtM26XZspG77Sjho67Tx9ucw1hAwg98xbPxqUcE6f1nH5U): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.736929240 (22h6m38.520269339s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737040080 (22h6m38.520380179s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SUcotB1144nMnt3JXXbVyNoz9aS6352uH9cGptt4RqKbvQkRv): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737054549 (22h6m38.520394678s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737102950 (22h6m38.520443049s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1272rh1tzNBuEmXqx4MpBe7NKXPac2wuNPCEpYqFk56gTrodcU2): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737108672 (22h6m38.520448771s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737196889 (22h6m38.520536988s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f1unaJrBkG1zw5XdBmCMeS8EKdqELYbQcr5cLq513xvctxcED): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737201459 (22h6m38.520541558s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737216567 (22h6m38.520556666s) [seg_pos={0 0}, upload=21163]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 00:10:52.737295508 (22h6m38.520635597s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 00:10:52.737299646 (22h6m38.520639745s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 00:11:00.530433246 (22h6m46.313773355s) [upload=21163]: Done uploading segments...
May 26 00:11:00.530455227 (22h6m46.313795347s) [upload=21163]: Upload finished. err: <nil>
Duration: 3.908244625s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
9 out of 60 is failed (threshold=3) [...........X...................X.......X.XXX...X........X..X] duration:
1m0.000775117s trace:
https://jaeger.dev.storj.io/trace/5b59e7787af4eb57
FAIL (2026-05-26 00:50:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000786308s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [......X..X..................................................] duration:
3.974407503s trace:
https://jaeger.dev.storj.io/trace/54dd3f55a77d17ef
FAIL (2026-05-26 00:49:53 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320420647 (22h6m37.103760746s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320702816 (22h6m37.104042915s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320993440 (22h6m37.104333539s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321256863 (22h6m37.104596962s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321827863 (22h6m37.105167962s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321916452 (22h6m37.105256551s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322280286 (22h6m37.105620385s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322452585 (22h6m37.105792684s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322950865 (22h6m37.106290964s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323536944 (22h6m37.106877043s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323731265 (22h6m37.107071364s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324123954 (22h6m37.107464053s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324414138 (22h6m37.107754227s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.325132458 (22h6m37.108472547s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.327055307 (22h6m37.110395406s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:52.388936548 (22h6m38.172276647s) [upload=21163]: Next returned nil segment
May 26 00:10:52.388980332 (22h6m38.172320431s) [upload=21163]: Waiting for error group managing segments...
May 26 00:10:52.674703801 (22h6m38.4580439s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.676576304 (22h6m38.459916403s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.685919140 (22h6m38.469259239s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.691062223 (22h6m38.474402322s) [noise=true , address=212.102.52.214:4347 , piece_id=OP75SIPBPG6SRO427YW5DXBI2LX3IGJYQ272EQABJCYXPJQSX3IA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.692733643 (22h6m38.476073742s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.695375413 (22h6m38.478715512s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.697960065 (22h6m38.481300164s) [noise=true , address=212.102.52.215:4390 , piece_id=E4XO5YC7J2INYSCZLMJO6VRV4DMDM4YDBYAOIHXR7NHCC5EZELKQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.699281385 (22h6m38.482621484s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.701163387 (22h6m38.484503486s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.702857559 (22h6m38.486197658s) [noise=true , address=212.102.52.214:4353 , piece_id=WZG7PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.705263059 (22h6m38.488603159s) [noise=true , address=212.102.52.215:4386 , piece_id=S5YIYIWUEMKONHJACR7E7PO7U7N2FMU3GH3OO4ULB77LLLC624MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707419504 (22h6m38.490759593s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707786013 (22h6m38.491126112s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.709872454 (22h6m38.493212543s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.730564756 (22h6m38.513904845s) [noise=true , address=212.102.52.215:4382 , piece_id=BAJ5RC7425LFHND3DAXX4YSESMOUIRZOH2SHWHITX4KT3OAE54YQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736735980 (22h6m38.520076079s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736752663 (22h6m38.520092762s) [seg_pos={0 0}, upload=21163]: Satisfied cohort requirements after 16 pieces
May 26 00:10:52.736921284 (22h6m38.520261383s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f4fqtM26XZspG77Sjho67Tx9ucw1hAwg98xbPxqUcE6f1nH5U): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.736929240 (22h6m38.520269339s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737040080 (22h6m38.520380179s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SUcotB1144nMnt3JXXbVyNoz9aS6352uH9cGptt4RqKbvQkRv): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737054549 (22h6m38.520394678s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737102950 (22h6m38.520443049s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1272rh1tzNBuEmXqx4MpBe7NKXPac2wuNPCEpYqFk56gTrodcU2): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737108672 (22h6m38.520448771s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737196889 (22h6m38.520536988s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f1unaJrBkG1zw5XdBmCMeS8EKdqELYbQcr5cLq513xvctxcED): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737201459 (22h6m38.520541558s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737216567 (22h6m38.520556666s) [seg_pos={0 0}, upload=21163]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 00:10:52.737295508 (22h6m38.520635597s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 00:10:52.737299646 (22h6m38.520639745s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 00:11:00.530433246 (22h6m46.313773355s) [upload=21163]: Done uploading segments...
May 26 00:11:00.530455227 (22h6m46.313795347s) [upload=21163]: Upload finished. err: <nil>
Duration: 3.974430878s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [.......X..X.................................................] duration:
6.869295926s trace:
https://jaeger.dev.storj.io/trace/54dd3f55a77d17ef
FAIL (2026-05-26 00:49:07 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320420647 (22h6m37.103760746s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320702816 (22h6m37.104042915s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320993440 (22h6m37.104333539s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321256863 (22h6m37.104596962s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321827863 (22h6m37.105167962s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321916452 (22h6m37.105256551s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322280286 (22h6m37.105620385s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322452585 (22h6m37.105792684s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322950865 (22h6m37.106290964s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323536944 (22h6m37.106877043s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323731265 (22h6m37.107071364s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324123954 (22h6m37.107464053s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324414138 (22h6m37.107754227s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.325132458 (22h6m37.108472547s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.327055307 (22h6m37.110395406s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:52.388936548 (22h6m38.172276647s) [upload=21163]: Next returned nil segment
May 26 00:10:52.388980332 (22h6m38.172320431s) [upload=21163]: Waiting for error group managing segments...
May 26 00:10:52.674703801 (22h6m38.4580439s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.676576304 (22h6m38.459916403s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.685919140 (22h6m38.469259239s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.691062223 (22h6m38.474402322s) [noise=true , address=212.102.52.214:4347 , piece_id=OP75SIPBPG6SRO427YW5DXBI2LX3IGJYQ272EQABJCYXPJQSX3IA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.692733643 (22h6m38.476073742s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.695375413 (22h6m38.478715512s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.697960065 (22h6m38.481300164s) [noise=true , address=212.102.52.215:4390 , piece_id=E4XO5YC7J2INYSCZLMJO6VRV4DMDM4YDBYAOIHXR7NHCC5EZELKQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.699281385 (22h6m38.482621484s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.701163387 (22h6m38.484503486s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.702857559 (22h6m38.486197658s) [noise=true , address=212.102.52.214:4353 , piece_id=WZG7PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.705263059 (22h6m38.488603159s) [noise=true , address=212.102.52.215:4386 , piece_id=S5YIYIWUEMKONHJACR7E7PO7U7N2FMU3GH3OO4ULB77LLLC624MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707419504 (22h6m38.490759593s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707786013 (22h6m38.491126112s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.709872454 (22h6m38.493212543s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.730564756 (22h6m38.513904845s) [noise=true , address=212.102.52.215:4382 , piece_id=BAJ5RC7425LFHND3DAXX4YSESMOUIRZOH2SHWHITX4KT3OAE54YQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736735980 (22h6m38.520076079s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736752663 (22h6m38.520092762s) [seg_pos={0 0}, upload=21163]: Satisfied cohort requirements after 16 pieces
May 26 00:10:52.736921284 (22h6m38.520261383s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f4fqtM26XZspG77Sjho67Tx9ucw1hAwg98xbPxqUcE6f1nH5U): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.736929240 (22h6m38.520269339s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737040080 (22h6m38.520380179s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SUcotB1144nMnt3JXXbVyNoz9aS6352uH9cGptt4RqKbvQkRv): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737054549 (22h6m38.520394678s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737102950 (22h6m38.520443049s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1272rh1tzNBuEmXqx4MpBe7NKXPac2wuNPCEpYqFk56gTrodcU2): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737108672 (22h6m38.520448771s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737196889 (22h6m38.520536988s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f1unaJrBkG1zw5XdBmCMeS8EKdqELYbQcr5cLq513xvctxcED): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737201459 (22h6m38.520541558s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737216567 (22h6m38.520556666s) [seg_pos={0 0}, upload=21163]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 00:10:52.737295508 (22h6m38.520635597s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 00:10:52.737299646 (22h6m38.520639745s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 00:11:00.530433246 (22h6m46.313773355s) [upload=21163]: Done uploading segments...
May 26 00:11:00.530455227 (22h6m46.313795347s) [upload=21163]: Upload finished. err: <nil>
Duration: 6.869315343s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [............X...................X.......X.XXX...X........X..] duration:
8.072814565s trace:
https://jaeger.dev.storj.io/trace/37caf16e5b5d88d5
FAIL (2026-05-26 00:48:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 8.072834514s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [.............X...................X.......X.XXX...X........X.] duration:
13.703258928s trace:
https://jaeger.dev.storj.io/trace/37caf16e5b5d88d5
FAIL (2026-05-26 00:48:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 13.703267736s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [..............X...................X.......X.XXX...X........X] duration:
1m0.00177238s trace:
https://jaeger.dev.storj.io/trace/37caf16e5b5d88d5
FAIL (2026-05-26 00:48:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001793179s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [........X..X................................................] duration:
4.603619159s trace:
https://jaeger.dev.storj.io/trace/54dd3f55a77d17ef
FAIL (2026-05-26 00:48:02 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320420647 (22h6m37.103760746s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320702816 (22h6m37.104042915s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320993440 (22h6m37.104333539s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321256863 (22h6m37.104596962s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321827863 (22h6m37.105167962s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321916452 (22h6m37.105256551s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322280286 (22h6m37.105620385s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322452585 (22h6m37.105792684s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322950865 (22h6m37.106290964s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323536944 (22h6m37.106877043s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323731265 (22h6m37.107071364s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324123954 (22h6m37.107464053s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324414138 (22h6m37.107754227s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.325132458 (22h6m37.108472547s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.327055307 (22h6m37.110395406s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:52.388936548 (22h6m38.172276647s) [upload=21163]: Next returned nil segment
May 26 00:10:52.388980332 (22h6m38.172320431s) [upload=21163]: Waiting for error group managing segments...
May 26 00:10:52.674703801 (22h6m38.4580439s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.676576304 (22h6m38.459916403s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.685919140 (22h6m38.469259239s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.691062223 (22h6m38.474402322s) [noise=true , address=212.102.52.214:4347 , piece_id=OP75SIPBPG6SRO427YW5DXBI2LX3IGJYQ272EQABJCYXPJQSX3IA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.692733643 (22h6m38.476073742s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.695375413 (22h6m38.478715512s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.697960065 (22h6m38.481300164s) [noise=true , address=212.102.52.215:4390 , piece_id=E4XO5YC7J2INYSCZLMJO6VRV4DMDM4YDBYAOIHXR7NHCC5EZELKQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.699281385 (22h6m38.482621484s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.701163387 (22h6m38.484503486s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.702857559 (22h6m38.486197658s) [noise=true , address=212.102.52.214:4353 , piece_id=WZG7PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.705263059 (22h6m38.488603159s) [noise=true , address=212.102.52.215:4386 , piece_id=S5YIYIWUEMKONHJACR7E7PO7U7N2FMU3GH3OO4ULB77LLLC624MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707419504 (22h6m38.490759593s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707786013 (22h6m38.491126112s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.709872454 (22h6m38.493212543s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.730564756 (22h6m38.513904845s) [noise=true , address=212.102.52.215:4382 , piece_id=BAJ5RC7425LFHND3DAXX4YSESMOUIRZOH2SHWHITX4KT3OAE54YQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736735980 (22h6m38.520076079s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736752663 (22h6m38.520092762s) [seg_pos={0 0}, upload=21163]: Satisfied cohort requirements after 16 pieces
May 26 00:10:52.736921284 (22h6m38.520261383s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f4fqtM26XZspG77Sjho67Tx9ucw1hAwg98xbPxqUcE6f1nH5U): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.736929240 (22h6m38.520269339s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737040080 (22h6m38.520380179s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SUcotB1144nMnt3JXXbVyNoz9aS6352uH9cGptt4RqKbvQkRv): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737054549 (22h6m38.520394678s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737102950 (22h6m38.520443049s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1272rh1tzNBuEmXqx4MpBe7NKXPac2wuNPCEpYqFk56gTrodcU2): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737108672 (22h6m38.520448771s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737196889 (22h6m38.520536988s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f1unaJrBkG1zw5XdBmCMeS8EKdqELYbQcr5cLq513xvctxcED): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737201459 (22h6m38.520541558s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737216567 (22h6m38.520556666s) [seg_pos={0 0}, upload=21163]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 00:10:52.737295508 (22h6m38.520635597s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 00:10:52.737299646 (22h6m38.520639745s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 00:11:00.530433246 (22h6m46.313773355s) [upload=21163]: Done uploading segments...
May 26 00:11:00.530455227 (22h6m46.313795347s) [upload=21163]: Upload finished. err: <nil>
Duration: 4.603642154s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [.........X..X...............................................] duration:
3.9431103s trace:
https://jaeger.dev.storj.io/trace/54dd3f55a77d17ef
FAIL (2026-05-26 00:47:38 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320420647 (22h6m37.103760746s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320702816 (22h6m37.104042915s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320993440 (22h6m37.104333539s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321256863 (22h6m37.104596962s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321827863 (22h6m37.105167962s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321916452 (22h6m37.105256551s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322280286 (22h6m37.105620385s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322452585 (22h6m37.105792684s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322950865 (22h6m37.106290964s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323536944 (22h6m37.106877043s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323731265 (22h6m37.107071364s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324123954 (22h6m37.107464053s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324414138 (22h6m37.107754227s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.325132458 (22h6m37.108472547s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.327055307 (22h6m37.110395406s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:52.388936548 (22h6m38.172276647s) [upload=21163]: Next returned nil segment
May 26 00:10:52.388980332 (22h6m38.172320431s) [upload=21163]: Waiting for error group managing segments...
May 26 00:10:52.674703801 (22h6m38.4580439s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.676576304 (22h6m38.459916403s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.685919140 (22h6m38.469259239s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.691062223 (22h6m38.474402322s) [noise=true , address=212.102.52.214:4347 , piece_id=OP75SIPBPG6SRO427YW5DXBI2LX3IGJYQ272EQABJCYXPJQSX3IA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.692733643 (22h6m38.476073742s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.695375413 (22h6m38.478715512s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.697960065 (22h6m38.481300164s) [noise=true , address=212.102.52.215:4390 , piece_id=E4XO5YC7J2INYSCZLMJO6VRV4DMDM4YDBYAOIHXR7NHCC5EZELKQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.699281385 (22h6m38.482621484s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.701163387 (22h6m38.484503486s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.702857559 (22h6m38.486197658s) [noise=true , address=212.102.52.214:4353 , piece_id=WZG7PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.705263059 (22h6m38.488603159s) [noise=true , address=212.102.52.215:4386 , piece_id=S5YIYIWUEMKONHJACR7E7PO7U7N2FMU3GH3OO4ULB77LLLC624MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707419504 (22h6m38.490759593s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707786013 (22h6m38.491126112s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.709872454 (22h6m38.493212543s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.730564756 (22h6m38.513904845s) [noise=true , address=212.102.52.215:4382 , piece_id=BAJ5RC7425LFHND3DAXX4YSESMOUIRZOH2SHWHITX4KT3OAE54YQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736735980 (22h6m38.520076079s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736752663 (22h6m38.520092762s) [seg_pos={0 0}, upload=21163]: Satisfied cohort requirements after 16 pieces
May 26 00:10:52.736921284 (22h6m38.520261383s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f4fqtM26XZspG77Sjho67Tx9ucw1hAwg98xbPxqUcE6f1nH5U): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.736929240 (22h6m38.520269339s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737040080 (22h6m38.520380179s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SUcotB1144nMnt3JXXbVyNoz9aS6352uH9cGptt4RqKbvQkRv): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737054549 (22h6m38.520394678s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737102950 (22h6m38.520443049s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1272rh1tzNBuEmXqx4MpBe7NKXPac2wuNPCEpYqFk56gTrodcU2): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737108672 (22h6m38.520448771s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737196889 (22h6m38.520536988s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f1unaJrBkG1zw5XdBmCMeS8EKdqELYbQcr5cLq513xvctxcED): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737201459 (22h6m38.520541558s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737216567 (22h6m38.520556666s) [seg_pos={0 0}, upload=21163]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 00:10:52.737295508 (22h6m38.520635597s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 00:10:52.737299646 (22h6m38.520639745s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 00:11:00.530433246 (22h6m46.313773355s) [upload=21163]: Done uploading segments...
May 26 00:11:00.530455227 (22h6m46.313795347s) [upload=21163]: Upload finished. err: <nil>
Duration: 3.943135358s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [..........X..X..............................................] duration:
4.103368619s trace:
https://jaeger.dev.storj.io/trace/54dd3f55a77d17ef
FAIL (2026-05-26 00:46:37 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320420647 (22h6m37.103760746s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320702816 (22h6m37.104042915s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320993440 (22h6m37.104333539s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321256863 (22h6m37.104596962s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321827863 (22h6m37.105167962s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321916452 (22h6m37.105256551s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322280286 (22h6m37.105620385s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322452585 (22h6m37.105792684s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322950865 (22h6m37.106290964s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323536944 (22h6m37.106877043s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323731265 (22h6m37.107071364s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324123954 (22h6m37.107464053s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324414138 (22h6m37.107754227s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.325132458 (22h6m37.108472547s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.327055307 (22h6m37.110395406s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:52.388936548 (22h6m38.172276647s) [upload=21163]: Next returned nil segment
May 26 00:10:52.388980332 (22h6m38.172320431s) [upload=21163]: Waiting for error group managing segments...
May 26 00:10:52.674703801 (22h6m38.4580439s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.676576304 (22h6m38.459916403s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.685919140 (22h6m38.469259239s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.691062223 (22h6m38.474402322s) [noise=true , address=212.102.52.214:4347 , piece_id=OP75SIPBPG6SRO427YW5DXBI2LX3IGJYQ272EQABJCYXPJQSX3IA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.692733643 (22h6m38.476073742s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.695375413 (22h6m38.478715512s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.697960065 (22h6m38.481300164s) [noise=true , address=212.102.52.215:4390 , piece_id=E4XO5YC7J2INYSCZLMJO6VRV4DMDM4YDBYAOIHXR7NHCC5EZELKQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.699281385 (22h6m38.482621484s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.701163387 (22h6m38.484503486s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.702857559 (22h6m38.486197658s) [noise=true , address=212.102.52.214:4353 , piece_id=WZG7PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.705263059 (22h6m38.488603159s) [noise=true , address=212.102.52.215:4386 , piece_id=S5YIYIWUEMKONHJACR7E7PO7U7N2FMU3GH3OO4ULB77LLLC624MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707419504 (22h6m38.490759593s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707786013 (22h6m38.491126112s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.709872454 (22h6m38.493212543s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.730564756 (22h6m38.513904845s) [noise=true , address=212.102.52.215:4382 , piece_id=BAJ5RC7425LFHND3DAXX4YSESMOUIRZOH2SHWHITX4KT3OAE54YQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736735980 (22h6m38.520076079s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736752663 (22h6m38.520092762s) [seg_pos={0 0}, upload=21163]: Satisfied cohort requirements after 16 pieces
May 26 00:10:52.736921284 (22h6m38.520261383s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f4fqtM26XZspG77Sjho67Tx9ucw1hAwg98xbPxqUcE6f1nH5U): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.736929240 (22h6m38.520269339s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737040080 (22h6m38.520380179s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SUcotB1144nMnt3JXXbVyNoz9aS6352uH9cGptt4RqKbvQkRv): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737054549 (22h6m38.520394678s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737102950 (22h6m38.520443049s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1272rh1tzNBuEmXqx4MpBe7NKXPac2wuNPCEpYqFk56gTrodcU2): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737108672 (22h6m38.520448771s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737196889 (22h6m38.520536988s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f1unaJrBkG1zw5XdBmCMeS8EKdqELYbQcr5cLq513xvctxcED): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737201459 (22h6m38.520541558s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737216567 (22h6m38.520556666s) [seg_pos={0 0}, upload=21163]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 00:10:52.737295508 (22h6m38.520635597s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 00:10:52.737299646 (22h6m38.520639745s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 00:11:00.530433246 (22h6m46.313773355s) [upload=21163]: Done uploading segments...
May 26 00:11:00.530455227 (22h6m46.313795347s) [upload=21163]: Upload finished. err: <nil>
Duration: 4.10338523s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [...............X...................X.......X.XXX...X........] duration:
9.577665946s trace:
https://jaeger.dev.storj.io/trace/74bd92de97cd3d42
FAIL (2026-05-26 00:46:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 9.577695904s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [X...........................................................] duration:
3.966455214s trace:
https://jaeger.dev.storj.io/trace/3bd7a1328937f878
FAIL (2026-05-26 00:46:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.966476846s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.X..........................................................] duration:
3.841931085s trace:
https://jaeger.dev.storj.io/trace/3bd7a1328937f878
FAIL (2026-05-26 00:45:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.841953559s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [................X...................X.......X.XXX...X.......] duration:
8.102114092s trace:
https://jaeger.dev.storj.io/trace/74bd92de97cd3d42
FAIL (2026-05-26 00:45:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 8.102132898s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [...........X..X.............................................] duration:
5.669833839s trace:
https://jaeger.dev.storj.io/trace/54dd3f55a77d17ef
FAIL (2026-05-26 00:45:41 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320420647 (22h6m37.103760746s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320702816 (22h6m37.104042915s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320993440 (22h6m37.104333539s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321256863 (22h6m37.104596962s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321827863 (22h6m37.105167962s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321916452 (22h6m37.105256551s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322280286 (22h6m37.105620385s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322452585 (22h6m37.105792684s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322950865 (22h6m37.106290964s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323536944 (22h6m37.106877043s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323731265 (22h6m37.107071364s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324123954 (22h6m37.107464053s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324414138 (22h6m37.107754227s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.325132458 (22h6m37.108472547s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.327055307 (22h6m37.110395406s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:52.388936548 (22h6m38.172276647s) [upload=21163]: Next returned nil segment
May 26 00:10:52.388980332 (22h6m38.172320431s) [upload=21163]: Waiting for error group managing segments...
May 26 00:10:52.674703801 (22h6m38.4580439s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.676576304 (22h6m38.459916403s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.685919140 (22h6m38.469259239s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.691062223 (22h6m38.474402322s) [noise=true , address=212.102.52.214:4347 , piece_id=OP75SIPBPG6SRO427YW5DXBI2LX3IGJYQ272EQABJCYXPJQSX3IA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.692733643 (22h6m38.476073742s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.695375413 (22h6m38.478715512s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.697960065 (22h6m38.481300164s) [noise=true , address=212.102.52.215:4390 , piece_id=E4XO5YC7J2INYSCZLMJO6VRV4DMDM4YDBYAOIHXR7NHCC5EZELKQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.699281385 (22h6m38.482621484s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.701163387 (22h6m38.484503486s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.702857559 (22h6m38.486197658s) [noise=true , address=212.102.52.214:4353 , piece_id=WZG7PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.705263059 (22h6m38.488603159s) [noise=true , address=212.102.52.215:4386 , piece_id=S5YIYIWUEMKONHJACR7E7PO7U7N2FMU3GH3OO4ULB77LLLC624MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707419504 (22h6m38.490759593s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707786013 (22h6m38.491126112s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.709872454 (22h6m38.493212543s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.730564756 (22h6m38.513904845s) [noise=true , address=212.102.52.215:4382 , piece_id=BAJ5RC7425LFHND3DAXX4YSESMOUIRZOH2SHWHITX4KT3OAE54YQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736735980 (22h6m38.520076079s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736752663 (22h6m38.520092762s) [seg_pos={0 0}, upload=21163]: Satisfied cohort requirements after 16 pieces
May 26 00:10:52.736921284 (22h6m38.520261383s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f4fqtM26XZspG77Sjho67Tx9ucw1hAwg98xbPxqUcE6f1nH5U): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.736929240 (22h6m38.520269339s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737040080 (22h6m38.520380179s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SUcotB1144nMnt3JXXbVyNoz9aS6352uH9cGptt4RqKbvQkRv): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737054549 (22h6m38.520394678s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737102950 (22h6m38.520443049s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1272rh1tzNBuEmXqx4MpBe7NKXPac2wuNPCEpYqFk56gTrodcU2): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737108672 (22h6m38.520448771s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737196889 (22h6m38.520536988s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f1unaJrBkG1zw5XdBmCMeS8EKdqELYbQcr5cLq513xvctxcED): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737201459 (22h6m38.520541558s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737216567 (22h6m38.520556666s) [seg_pos={0 0}, upload=21163]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 00:10:52.737295508 (22h6m38.520635597s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 00:10:52.737299646 (22h6m38.520639745s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 00:11:00.530433246 (22h6m46.313773355s) [upload=21163]: Done uploading segments...
May 26 00:11:00.530455227 (22h6m46.313795347s) [upload=21163]: Upload finished. err: <nil>
Duration: 5.669867633s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [............X..X............................................] duration:
4.208123057s trace:
https://jaeger.dev.storj.io/trace/54dd3f55a77d17ef
FAIL (2026-05-26 00:45:21 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320420647 (22h6m37.103760746s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320702816 (22h6m37.104042915s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320993440 (22h6m37.104333539s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321256863 (22h6m37.104596962s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321827863 (22h6m37.105167962s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321916452 (22h6m37.105256551s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322280286 (22h6m37.105620385s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322452585 (22h6m37.105792684s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322950865 (22h6m37.106290964s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323536944 (22h6m37.106877043s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323731265 (22h6m37.107071364s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324123954 (22h6m37.107464053s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324414138 (22h6m37.107754227s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.325132458 (22h6m37.108472547s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.327055307 (22h6m37.110395406s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:52.388936548 (22h6m38.172276647s) [upload=21163]: Next returned nil segment
May 26 00:10:52.388980332 (22h6m38.172320431s) [upload=21163]: Waiting for error group managing segments...
May 26 00:10:52.674703801 (22h6m38.4580439s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.676576304 (22h6m38.459916403s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.685919140 (22h6m38.469259239s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.691062223 (22h6m38.474402322s) [noise=true , address=212.102.52.214:4347 , piece_id=OP75SIPBPG6SRO427YW5DXBI2LX3IGJYQ272EQABJCYXPJQSX3IA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.692733643 (22h6m38.476073742s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.695375413 (22h6m38.478715512s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.697960065 (22h6m38.481300164s) [noise=true , address=212.102.52.215:4390 , piece_id=E4XO5YC7J2INYSCZLMJO6VRV4DMDM4YDBYAOIHXR7NHCC5EZELKQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.699281385 (22h6m38.482621484s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.701163387 (22h6m38.484503486s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.702857559 (22h6m38.486197658s) [noise=true , address=212.102.52.214:4353 , piece_id=WZG7PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.705263059 (22h6m38.488603159s) [noise=true , address=212.102.52.215:4386 , piece_id=S5YIYIWUEMKONHJACR7E7PO7U7N2FMU3GH3OO4ULB77LLLC624MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707419504 (22h6m38.490759593s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707786013 (22h6m38.491126112s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.709872454 (22h6m38.493212543s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.730564756 (22h6m38.513904845s) [noise=true , address=212.102.52.215:4382 , piece_id=BAJ5RC7425LFHND3DAXX4YSESMOUIRZOH2SHWHITX4KT3OAE54YQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736735980 (22h6m38.520076079s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736752663 (22h6m38.520092762s) [seg_pos={0 0}, upload=21163]: Satisfied cohort requirements after 16 pieces
May 26 00:10:52.736921284 (22h6m38.520261383s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f4fqtM26XZspG77Sjho67Tx9ucw1hAwg98xbPxqUcE6f1nH5U): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.736929240 (22h6m38.520269339s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737040080 (22h6m38.520380179s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SUcotB1144nMnt3JXXbVyNoz9aS6352uH9cGptt4RqKbvQkRv): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737054549 (22h6m38.520394678s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737102950 (22h6m38.520443049s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1272rh1tzNBuEmXqx4MpBe7NKXPac2wuNPCEpYqFk56gTrodcU2): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737108672 (22h6m38.520448771s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737196889 (22h6m38.520536988s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f1unaJrBkG1zw5XdBmCMeS8EKdqELYbQcr5cLq513xvctxcED): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737201459 (22h6m38.520541558s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737216567 (22h6m38.520556666s) [seg_pos={0 0}, upload=21163]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 00:10:52.737295508 (22h6m38.520635597s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 00:10:52.737299646 (22h6m38.520639745s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 00:11:00.530433246 (22h6m46.313773355s) [upload=21163]: Done uploading segments...
May 26 00:11:00.530455227 (22h6m46.313795347s) [upload=21163]: Upload finished. err: <nil>
Duration: 4.208148215s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [.................X...................X.......X.XXX...X......] duration:
17.508012785s trace:
https://jaeger.dev.storj.io/trace/74bd92de97cd3d42
FAIL (2026-05-26 00:44:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 17.508035999s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..X.........................................................] duration:
3.752724513s trace:
https://jaeger.dev.storj.io/trace/3bd7a1328937f878
FAIL (2026-05-26 00:44:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.752734754s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...X........................................................] duration:
3.559660464s trace:
https://jaeger.dev.storj.io/trace/3bd7a1328937f878
FAIL (2026-05-26 00:44:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.559684279s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [.............X..X...........................................] duration:
5.181617728s trace:
https://jaeger.dev.storj.io/trace/54dd3f55a77d17ef
FAIL (2026-05-26 00:44:09 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320420647 (22h6m37.103760746s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320702816 (22h6m37.104042915s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320993440 (22h6m37.104333539s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321256863 (22h6m37.104596962s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321827863 (22h6m37.105167962s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321916452 (22h6m37.105256551s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322280286 (22h6m37.105620385s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322452585 (22h6m37.105792684s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322950865 (22h6m37.106290964s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323536944 (22h6m37.106877043s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323731265 (22h6m37.107071364s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324123954 (22h6m37.107464053s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324414138 (22h6m37.107754227s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.325132458 (22h6m37.108472547s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.327055307 (22h6m37.110395406s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:52.388936548 (22h6m38.172276647s) [upload=21163]: Next returned nil segment
May 26 00:10:52.388980332 (22h6m38.172320431s) [upload=21163]: Waiting for error group managing segments...
May 26 00:10:52.674703801 (22h6m38.4580439s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.676576304 (22h6m38.459916403s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.685919140 (22h6m38.469259239s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.691062223 (22h6m38.474402322s) [noise=true , address=212.102.52.214:4347 , piece_id=OP75SIPBPG6SRO427YW5DXBI2LX3IGJYQ272EQABJCYXPJQSX3IA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.692733643 (22h6m38.476073742s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.695375413 (22h6m38.478715512s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.697960065 (22h6m38.481300164s) [noise=true , address=212.102.52.215:4390 , piece_id=E4XO5YC7J2INYSCZLMJO6VRV4DMDM4YDBYAOIHXR7NHCC5EZELKQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.699281385 (22h6m38.482621484s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.701163387 (22h6m38.484503486s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.702857559 (22h6m38.486197658s) [noise=true , address=212.102.52.214:4353 , piece_id=WZG7PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.705263059 (22h6m38.488603159s) [noise=true , address=212.102.52.215:4386 , piece_id=S5YIYIWUEMKONHJACR7E7PO7U7N2FMU3GH3OO4ULB77LLLC624MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707419504 (22h6m38.490759593s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707786013 (22h6m38.491126112s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.709872454 (22h6m38.493212543s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.730564756 (22h6m38.513904845s) [noise=true , address=212.102.52.215:4382 , piece_id=BAJ5RC7425LFHND3DAXX4YSESMOUIRZOH2SHWHITX4KT3OAE54YQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736735980 (22h6m38.520076079s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736752663 (22h6m38.520092762s) [seg_pos={0 0}, upload=21163]: Satisfied cohort requirements after 16 pieces
May 26 00:10:52.736921284 (22h6m38.520261383s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f4fqtM26XZspG77Sjho67Tx9ucw1hAwg98xbPxqUcE6f1nH5U): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.736929240 (22h6m38.520269339s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737040080 (22h6m38.520380179s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SUcotB1144nMnt3JXXbVyNoz9aS6352uH9cGptt4RqKbvQkRv): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737054549 (22h6m38.520394678s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737102950 (22h6m38.520443049s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1272rh1tzNBuEmXqx4MpBe7NKXPac2wuNPCEpYqFk56gTrodcU2): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737108672 (22h6m38.520448771s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737196889 (22h6m38.520536988s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f1unaJrBkG1zw5XdBmCMeS8EKdqELYbQcr5cLq513xvctxcED): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737201459 (22h6m38.520541558s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737216567 (22h6m38.520556666s) [seg_pos={0 0}, upload=21163]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 00:10:52.737295508 (22h6m38.520635597s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 00:10:52.737299646 (22h6m38.520639745s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 00:11:00.530433246 (22h6m46.313773355s) [upload=21163]: Done uploading segments...
May 26 00:11:00.530455227 (22h6m46.313795347s) [upload=21163]: Upload finished. err: <nil>
Duration: 5.18163968s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [..................X...................X.......X.XXX...X.....] duration:
6.498483703s trace:
https://jaeger.dev.storj.io/trace/74bd92de97cd3d42
FAIL (2026-05-26 00:44:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.498495124s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [....X.......................................................] duration:
4.637716471s trace:
https://jaeger.dev.storj.io/trace/3bd7a1328937f878
FAIL (2026-05-26 00:43:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.637740587s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [..............X..X..........................................] duration:
4.062071471s trace:
https://jaeger.dev.storj.io/trace/54dd3f55a77d17ef
FAIL (2026-05-26 00:43:15 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320420647 (22h6m37.103760746s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320702816 (22h6m37.104042915s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.320993440 (22h6m37.104333539s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321256863 (22h6m37.104596962s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321827863 (22h6m37.105167962s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.321916452 (22h6m37.105256551s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322280286 (22h6m37.105620385s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322452585 (22h6m37.105792684s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.322950865 (22h6m37.106290964s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323536944 (22h6m37.106877043s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.323731265 (22h6m37.107071364s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324123954 (22h6m37.107464053s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.324414138 (22h6m37.107754227s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.325132458 (22h6m37.108472547s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:51.327055307 (22h6m37.110395406s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Uploading piece...
May 26 00:10:52.388936548 (22h6m38.172276647s) [upload=21163]: Next returned nil segment
May 26 00:10:52.388980332 (22h6m38.172320431s) [upload=21163]: Waiting for error group managing segments...
May 26 00:10:52.674703801 (22h6m38.4580439s) [noise=true , address=212.102.52.216:4399 , piece_id=K5B3VDSSOJYPKIQWSZA3LAIP6ECGX4UIYTLHJKLEV65EEROPAXWA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.676576304 (22h6m38.459916403s) [noise=true , address=212.102.52.214:4358 , piece_id=XYDUBDN62GKNQB3O5H2BWWROODBZKXNJVEZRXCYXTRE3XGTKPCZQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.685919140 (22h6m38.469259239s) [noise=true , address=212.102.52.215:4371 , piece_id=OJPOOTWXGORXQYWWFFOKOT33F6KJI4A2355TNSMYE4OQXRU3GM3Q, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.691062223 (22h6m38.474402322s) [noise=true , address=212.102.52.214:4347 , piece_id=OP75SIPBPG6SRO427YW5DXBI2LX3IGJYQ272EQABJCYXPJQSX3IA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.692733643 (22h6m38.476073742s) [noise=true , address=212.102.52.213:4341 , piece_id=IZ7SWWTG5TQT5L725QJU7SBWBYQ4HW2R6C4DO7C2LWEJL7UMKX3A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.695375413 (22h6m38.478715512s) [noise=true , address=212.102.52.213:4324 , piece_id=IHZCGYVL5XVHHQBTQGNXJKVDNCKGLMPPI7CMP5U5QSEC2KBKS2ZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.697960065 (22h6m38.481300164s) [noise=true , address=212.102.52.215:4390 , piece_id=E4XO5YC7J2INYSCZLMJO6VRV4DMDM4YDBYAOIHXR7NHCC5EZELKQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.699281385 (22h6m38.482621484s) [noise=true , address=212.102.52.216:4402 , piece_id=YIB3T4SMRTQJGTAVUESYSEKOEUGWYNSUUIE36NTNYCXJGBS6GZJA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.701163387 (22h6m38.484503486s) [noise=true , address=212.102.52.214:4367 , piece_id=P25PLZRCWMM5F2ZV2M6AE6KZH23PDQWYCC2KWJORUO55RG2BKAZA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.702857559 (22h6m38.486197658s) [noise=true , address=212.102.52.214:4353 , piece_id=WZG7PJO5RODWWW63G26AKA5ZAP4LUVF66AOOKRIIBUUO4IWP7GOA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.705263059 (22h6m38.488603159s) [noise=true , address=212.102.52.215:4386 , piece_id=S5YIYIWUEMKONHJACR7E7PO7U7N2FMU3GH3OO4ULB77LLLC624MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707419504 (22h6m38.490759593s) [noise=true , address=212.102.52.217:4433 , piece_id=IBUBF2KSK4XJJY3GGUU7ELHZA6QS7LENZYUKXAA24NYSSAYJQAXQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.707786013 (22h6m38.491126112s) [noise=true , address=212.102.52.217:4420 , piece_id=JDI5NNY6OJUU2DSIFYDTDVYJD2POJDFPAD3HA237DZ43E2R2MYHQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.709872454 (22h6m38.493212543s) [noise=true , address=212.102.52.216:4407 , piece_id=DWZNVQEGJ2V34UDSVXLGGBHLKVCF4QYTWLYJ5GWUUVWGVUQCDSVA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.730564756 (22h6m38.513904845s) [noise=true , address=212.102.52.215:4382 , piece_id=BAJ5RC7425LFHND3DAXX4YSESMOUIRZOH2SHWHITX4KT3OAE54YQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736735980 (22h6m38.520076079s) [noise=true , address=212.102.52.217:4431 , piece_id=P7SKOTYTIASYYKMEFKYQTDJ3ZP2CVYOYATQPEYQDG2P65ZUOPWQA, seg_pos={0 0}, upload=21163]: Done uploading piece. err: <nil>
May 26 00:10:52.736752663 (22h6m38.520092762s) [seg_pos={0 0}, upload=21163]: Satisfied cohort requirements after 16 pieces
May 26 00:10:52.736921284 (22h6m38.520261383s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f4fqtM26XZspG77Sjho67Tx9ucw1hAwg98xbPxqUcE6f1nH5U): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.736929240 (22h6m38.520269339s) [noise=true , address=212.102.52.217:4424 , piece_id=56YU26BQTQY7FJUHZ7T5ISAPEYY7QAJI2Q3Z6J2MDAKYBVYACJ7A, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737040080 (22h6m38.520380179s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SUcotB1144nMnt3JXXbVyNoz9aS6352uH9cGptt4RqKbvQkRv): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737054549 (22h6m38.520394678s) [noise=true , address=212.102.52.213:4339 , piece_id=MN3JN37HKGKLVKJYXMXVSMITKU4PCPH6NDD7Z5U37YZ62ZS2U4TQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737102950 (22h6m38.520443049s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1272rh1tzNBuEmXqx4MpBe7NKXPac2wuNPCEpYqFk56gTrodcU2): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737108672 (22h6m38.520448771s) [noise=true , address=212.102.52.216:4405 , piece_id=OXJTIEZ2HUNIGFLRCEM64MELMI75XZUXRMQLM4EYFIWZWQIXN7MQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737196889 (22h6m38.520536988s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12f1unaJrBkG1zw5XdBmCMeS8EKdqELYbQcr5cLq513xvctxcED): protocol: expected piece hash; context canceled; EOF
May 26 00:10:52.737201459 (22h6m38.520541558s) [noise=true , address=212.102.52.213:4322 , piece_id=KHVYCOXFDTYTYWSGQBU65FZBVLULQZW25PXAXDTGVDBCREPSKZUQ, seg_pos={0 0}, upload=21163]: Long tail cancellation due to optimal threshold
May 26 00:10:52.737216567 (22h6m38.520556666s) [seg_pos={0 0}, upload=21163]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 26 00:10:52.737295508 (22h6m38.520635597s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 26 00:10:52.737299646 (22h6m38.520639745s) [upload=21163]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 26 00:11:00.530433246 (22h6m46.313773355s) [upload=21163]: Done uploading segments...
May 26 00:11:00.530455227 (22h6m46.313795347s) [upload=21163]: Upload finished. err: <nil>
Duration: 4.06209254s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [X..................X...................X.......X.XXX...X....] duration:
6.488224245s trace:
https://jaeger.dev.storj.io/trace/74bd92de97cd3d42
FAIL (2026-05-26 00:43:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.488240165s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.....X......................................................] duration:
3.606019934s trace:
https://jaeger.dev.storj.io/trace/3bd7a1328937f878
FAIL (2026-05-26 00:42:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.606041094s
core
uplink
us-select-1
us1-placement-20
|