Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.........X...X...X...........................X.........XX...] duration:
20.262107469s trace:
https://jaeger.dev.storj.io/trace/5d9345029f8e93d5
FAIL (2026-05-25 19:12:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 20.26211827s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [..........X...X...X...........................X.........XX..] duration:
32.183744119s trace:
https://jaeger.dev.storj.io/trace/5d9345029f8e93d5
FAIL (2026-05-25 19:12:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 32.18375549s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...............X.......................X....................] duration:
3.900295352s trace:
https://jaeger.dev.storj.io/trace/1a1cdbfbe5dd42e6
FAIL (2026-05-25 19:12:17 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.900316812s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [................X.......................X...................] duration:
3.963679846s trace:
https://jaeger.dev.storj.io/trace/1a1cdbfbe5dd42e6
FAIL (2026-05-25 19:11:40 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.963700907s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [...........X...X...X...........................X.........XX.] duration:
28.27211018s trace:
https://jaeger.dev.storj.io/trace/5d9345029f8e93d5
FAIL (2026-05-25 19:11:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 28.272119298s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.................X.......................X..................] duration:
3.700094167s trace:
https://jaeger.dev.storj.io/trace/1a1cdbfbe5dd42e6
FAIL (2026-05-25 19:11:00 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.700118363s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [X...........X...X...X...........................X.........XX] duration:
1m0.001132092s trace:
https://jaeger.dev.storj.io/trace/5d9345029f8e93d5
FAIL (2026-05-25 19:10:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001139474s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..................X.......................X.................] duration:
3.957901839s trace:
https://jaeger.dev.storj.io/trace/1a1cdbfbe5dd42e6
FAIL (2026-05-25 19:10:39 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.957927607s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...................X.......................X................] duration:
4.248841379s trace:
https://jaeger.dev.storj.io/trace/1a1cdbfbe5dd42e6
FAIL (2026-05-25 19:09:56 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.248860575s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [....................X.......................X...............] duration:
4.237371893s trace:
https://jaeger.dev.storj.io/trace/1a1cdbfbe5dd42e6
FAIL (2026-05-25 19:09:41 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.237388454s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
6 out of 60 is failed (threshold=3) [.X...........X...X...X...........................X.........X] duration:
1m0.000521465s trace:
https://jaeger.dev.storj.io/trace/7166c504ce6e64
FAIL (2026-05-25 19:09:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000592571s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
3.916888022s trace:
https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1
FAIL (2026-05-25 19:09:13 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
77]: Uploading piece...
May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment
May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments...
May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces
May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments...
May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded
Duration: 3.916910416s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
4.027655899s trace:
https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1
FAIL (2026-05-25 19:08:42 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
77]: Uploading piece...
May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment
May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments...
May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces
May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments...
May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded
Duration: 4.027677901s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.....................X.......................X..............] duration:
3.879029753s trace:
https://jaeger.dev.storj.io/trace/1a1cdbfbe5dd42e6
FAIL (2026-05-25 19:08:31 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.879049782s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [......................X.......................X.............] duration:
5.001363344s trace:
https://jaeger.dev.storj.io/trace/1a1cdbfbe5dd42e6
FAIL (2026-05-25 19:08:16 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001385454s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [..X...........X...X...X...........................X.........] duration:
6.652740729s trace:
https://jaeger.dev.storj.io/trace/17709949c5164ba
FAIL (2026-05-25 19:08:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.652992189s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
4.052676377s trace:
https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1
FAIL (2026-05-25 19:07:28 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
77]: Uploading piece...
May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment
May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments...
May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces
May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments...
May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded
Duration: 4.052696525s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.......................X.......................X............] duration:
3.752499877s trace:
https://jaeger.dev.storj.io/trace/1a1cdbfbe5dd42e6
FAIL (2026-05-25 19:07:25 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.752513322s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [...X...........X...X...X...........................X........] duration:
6.836540658s trace:
https://jaeger.dev.storj.io/trace/17709949c5164ba
FAIL (2026-05-25 19:07:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.836583569s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [X..X........................................................] duration:
3.972611362s trace:
https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1
FAIL (2026-05-25 19:06:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
77]: Uploading piece...
May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment
May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments...
May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces
May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments...
May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded
Duration: 3.972644806s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [....X...........X...X...X...........................X.......] duration:
6.902171861s trace:
https://jaeger.dev.storj.io/trace/17709949c5164ba
FAIL (2026-05-25 19:06:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.902199854s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [.X..X.......................................................] duration:
4.056584365s trace:
https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1
FAIL (2026-05-25 19:06:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
77]: Uploading piece...
May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment
May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments...
May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces
May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments...
May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded
Duration: 4.056599756s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [........................X.......................X...........] duration:
3.809661019s trace:
https://jaeger.dev.storj.io/trace/1a1cdbfbe5dd42e6
FAIL (2026-05-25 19:06:15 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.809684743s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.........................X.......................X..........] duration:
3.81303461s trace:
https://jaeger.dev.storj.io/trace/1a1cdbfbe5dd42e6
FAIL (2026-05-25 19:05:42 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.813059979s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.....X...........X...X...X...........................X......] duration:
7.13818876s trace:
https://jaeger.dev.storj.io/trace/17709949c5164ba
FAIL (2026-05-25 19:05:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.138218887s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [..X..X......................................................] duration:
3.944909906s trace:
https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1
FAIL (2026-05-25 19:05:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
77]: Uploading piece...
May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment
May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments...
May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces
May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments...
May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded
Duration: 3.944932378s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..........................X.......................X.........] duration:
3.851016161s trace:
https://jaeger.dev.storj.io/trace/1a1cdbfbe5dd42e6
FAIL (2026-05-25 19:04:53 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.851059194s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [...X..X.....................................................] duration:
4.205116059s trace:
https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1
FAIL (2026-05-25 19:04:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
77]: Uploading piece...
May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment
May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments...
May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces
May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments...
May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded
Duration: 4.205140074s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [......X...........X...X...X...........................X.....] duration:
6.791138173s trace:
https://jaeger.dev.storj.io/trace/17709949c5164ba
FAIL (2026-05-25 19:04:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.791156168s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.......X...........X...X...X...........................X....] duration:
6.901510299s trace:
https://jaeger.dev.storj.io/trace/17709949c5164ba
FAIL (2026-05-25 19:03:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.901532601s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...........................X.......................X........] duration:
3.806302528s trace:
https://jaeger.dev.storj.io/trace/1a1cdbfbe5dd42e6
FAIL (2026-05-25 19:03:44 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.806318498s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [....X..X....................................................] duration:
4.2048444s trace:
https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1
FAIL (2026-05-25 19:03:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
77]: Uploading piece...
May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment
May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments...
May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces
May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments...
May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded
Duration: 4.20486572s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [........X...........X...X...X...........................X...] duration:
6.473918106s trace:
https://jaeger.dev.storj.io/trace/17709949c5164ba
FAIL (2026-05-25 19:03:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.473929096s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.........X...........X...X...X...........................X..] duration:
33.858784133s trace:
https://jaeger.dev.storj.io/trace/17709949c5164ba
FAIL (2026-05-25 19:03:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 33.858803439s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [............................X.......................X.......] duration:
3.94703482s trace:
https://jaeger.dev.storj.io/trace/1a1cdbfbe5dd42e6
FAIL (2026-05-25 19:03:15 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.947056371s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [.....X..X...................................................] duration:
4.062789137s trace:
https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1
FAIL (2026-05-25 19:02:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
77]: Uploading piece...
May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment
May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments...
May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces
May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments...
May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded
Duration: 4.06281235s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [..........X...........X...X...X...........................X.] duration:
6.937194358s trace:
https://jaeger.dev.storj.io/trace/17709949c5164ba
FAIL (2026-05-25 19:02:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.937203766s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [...........X...........X...X...X...........................X] duration:
1m0.001396596s trace:
https://jaeger.dev.storj.io/trace/17709949c5164ba
FAIL (2026-05-25 19:02:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001417115s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.............................X.......................X......] duration:
4.487883862s trace:
https://jaeger.dev.storj.io/trace/1a1cdbfbe5dd42e6
FAIL (2026-05-25 19:02:13 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.487902578s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..............................X.......................X.....] duration:
3.979700769s trace:
https://jaeger.dev.storj.io/trace/1a1cdbfbe5dd42e6
FAIL (2026-05-25 19:01:22 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.979745596s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [......X..X..................................................] duration:
4.171747034s trace:
https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1
FAIL (2026-05-25 19:01:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
77]: Uploading piece...
May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment
May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments...
May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces
May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments...
May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded
Duration: 4.171757763s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...............................X.......................X....] duration:
3.872686913s trace:
https://jaeger.dev.storj.io/trace/1a1cdbfbe5dd42e6
FAIL (2026-05-25 19:00:17 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.872704616s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [X...........X...........X...X...X...........................] duration:
9.842835495s trace:
https://jaeger.dev.storj.io/trace/4fc3ac9cb273d912
FAIL (2026-05-25 19:00:16 UTC)
orbiter warning: context deadline exceeded
Duration: 9.842858809s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [.......X..X.................................................] duration:
4.929789367s trace:
https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1
FAIL (2026-05-25 19:00:08 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
77]: Uploading piece...
May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment
May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments...
May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces
May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments...
May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded
Duration: 4.929812501s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.X...........X...........X...X...X..........................] duration:
6.952701104s trace:
https://jaeger.dev.storj.io/trace/4fc3ac9cb273d912
FAIL (2026-05-25 18:59:32 UTC)
orbiter warning: context deadline exceeded
Duration: 6.952712395s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [..X...........X...........X...X...X.........................] duration:
36.801375558s trace:
https://jaeger.dev.storj.io/trace/4fc3ac9cb273d912
FAIL (2026-05-25 18:59:25 UTC)
orbiter warning: context deadline exceeded
Duration: 36.801425062s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [................................X.......................X...] duration:
3.761075808s trace:
https://jaeger.dev.storj.io/trace/1a1cdbfbe5dd42e6
FAIL (2026-05-25 18:59:11 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.761094303s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [........X..X................................................] duration:
3.840098664s trace:
https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1
FAIL (2026-05-25 18:58:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
77]: Uploading piece...
May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment
May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments...
May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces
May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments...
May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded
Duration: 3.840119715s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [...X...........X...........X...X...X........................] duration:
17.498384803s trace:
https://jaeger.dev.storj.io/trace/4fc3ac9cb273d912
FAIL (2026-05-25 18:58:47 UTC)
orbiter warning: context deadline exceeded
Duration: 17.498404872s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [.........X..X...............................................] duration:
3.999471746s trace:
https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1
FAIL (2026-05-25 18:58:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
77]: Uploading piece...
May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment
May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments...
May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces
May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments...
May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded
Duration: 3.999496204s
core
uplink
uk-select-1
uk1-placement-17
|