Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
4 out of 60 is failed (threshold=3) [.......................X................X...............X..X] duration:
1m40.714762271s trace:
https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1
FAIL (2026-05-25 18:22:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
77]: Uploading piece...
May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece...
May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment
May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments...
May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil>
May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces
May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF
May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold
May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments...
May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded
Duration: 1m40.714773784s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
13 out of 60 is failed (threshold=3) [...X....X.......X......X..XX.XX.X....X.....XX......X........] duration:
9.163159063s trace:
https://jaeger.dev.storj.io/trace/698dfada1f99867a
FAIL (2026-05-25 18:21:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 9.163182888s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.............................X..............................] duration:
3.834336802s trace:
https://jaeger.dev.storj.io/trace/14eb8131f680ed75
FAIL (2026-05-25 18:21:29 UTC)
orbiter warning: context deadline exceeded
Duration: 3.834354685s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
678.968248ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/7e710747942acdb2
FAIL (2026-05-25 18:21:17 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 678.989068ms
core
satellite
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
13 out of 60 is failed (threshold=3) [....X....X.......X......X..XX.XX.X....X.....XX......X.......] duration:
7.01082867s trace:
https://jaeger.dev.storj.io/trace/698dfada1f99867a
FAIL (2026-05-25 18:21:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.010850422s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
676.231791ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/7e710747942acdb2
FAIL (2026-05-25 18:20:26 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 676.263421ms
core
satellite
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..............................X.............................] duration:
4.081093877s trace:
https://jaeger.dev.storj.io/trace/14eb8131f680ed75
FAIL (2026-05-25 18:20:17 UTC)
orbiter warning: context deadline exceeded
Duration: 4.081107624s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
13 out of 60 is failed (threshold=3) [.....X....X.......X......X..XX.XX.X....X.....XX......X......] duration:
7.005772803s trace:
https://jaeger.dev.storj.io/trace/698dfada1f99867a
FAIL (2026-05-25 18:20:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.005794124s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
678.919799ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/7e710747942acdb2
FAIL (2026-05-25 18:19:46 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 678.945558ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
3 out of 60 is failed (threshold=3) [........................X................X...............X..] duration:
29.830636029s trace:
https://jaeger.dev.storj.io/trace/47c8961ca375d003
FAIL (2026-05-25 18:19:44 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
DVKEJ2Q, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.826862166 (16h13m52.610202265s) [noise=true , address=212.102.52.216:4410 , piece_id=CNVT54RWTRYXAKSZW7SXKAFTJXJ7TJOMNMQV4SEZAWX6P25SMDAA, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.827074101 (16h13m52.6104142s) [noise=true , address=212.102.52.214:4359 , piece_id=QDTWWS3LMTJONF3G2GZE3OLYUCXUNMKAOWF3GCZXQHXEE2XWY5YQ, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.827499863 (16h13m52.610839962s) [noise=true , address=212.102.52.213:4331 , piece_id=EEG5YWPYZRZ5A55GPCGE7ZJCA3EOMYF6MJVKFOWTMDYPAZD7P2WQ, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.827663575 (16h13m52.611003674s) [noise=true , address=212.102.52.215:4376 , piece_id=MASYN2GTW4OGTF422XYQL2K4JIV633OFCQAT32ZHHBPUXDRZAHFA, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.827904234 (16h13m52.611244333s) [noise=true , address=212.102.52.217:4439 , piece_id=TOHCWZLYIUS7QE4N4ZAIB2QWUAVB2DE2SAZTG6OMQLWR5VVDDRNA, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.828316992 (16h13m52.611657091s) [noise=true , address=212.102.52.213:4323 , piece_id=VKBPFLZMPGTGYIRKDJ7CYJVIGRSBZ7BL4BRRYAPFOCXJZF3DHEGQ, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.828439305 (16h13m52.611779394s) [noise=true , address=212.102.52.213:4337 , piece_id=VWF4LCALY4RIE3KAFYNGANGUCHOCB7G7WS5IM2S2TXTT22ZCIICQ, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.828885316 (16h13m52.612225416s) [noise=true , address=212.102.52.213:4342 , piece_id=WNWW2PC5W7VYO5ZPOIY4GFUUEL2BNUDEBGMX53KXQQTRQAN744TA, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.829013611 (16h13m52.61235371s) [noise=true , address=212.102.52.214:4361 , piece_id=M53YWUBS6EJZK4IZWS6P7T3EJ7CPWXYL3CH7TRZW7CJZHQCUPXGA, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.829318423 (16h13m52.612658522s) [noise=true , address=212.102.52.214:4368 , piece_id=DFPNMKRLFN43RCYCIO5TMOSIVWLMBJPNPZMO5BZ3AEKDC5SARLQA, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.829676847 (16h13m52.613016946s) [noise=true , address=212.102.52.217:4432 , piece_id=UF3ANPTDI6H3Z4IG3HQIDLCA4J32MKTXAND6XYJ2HNT22OKMGDHQ, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.829790153 (16h13m52.613130252s) [noise=true , address=212.102.52.215:4389 , piece_id=FDYCELU27TEDNDALWQ565K3AE5FVUMGLOIQ3PUIIVIXUAMYPUKHA, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.830404054 (16h13m52.613744153s) [noise=true , address=212.102.52.214:4364 , piece_id=IISEDBNGQ7EW5BWECJ5M7E56YP5XAA5VP3JEMYFPMRYHE2KN3HXQ, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.831055047 (16h13m52.614395146s) [noise=true , address=212.102.52.215:4381 , piece_id=WAPMAH3XPQPJVQUOZBNONWMFCT7T3LGFX7BMTHMAPU4PIZAMQ2MA, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.831772195 (16h13m52.615112304s) [noise=true , address=212.102.52.215:4384 , piece_id=R5RDE456F34SAVPUBICPDJGAXHYOOH4ILTDIBAKLGL34PMMFVYQA, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:07.900796750 (16h13m53.68413685s) [upload=15534]: Next returned nil segment
May 25 18:18:07.900813633 (16h13m53.684153722s) [upload=15534]: Waiting for error group managing segments...
May 25 18:18:08.180403392 (16h13m53.963743492s) [noise=true , address=212.102.52.213:4323 , piece_id=VKBPFLZMPGTGYIRKDJ7CYJVIGRSBZ7BL4BRRYAPFOCXJZF3DHEGQ, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.185311568 (16h13m53.968651657s) [noise=true , address=212.102.52.217:4419 , piece_id=JYS43OLSNC3HGNDCICQHGZ4NV5PDKSP4PZGVQYGT7LFJXDVLDFEQ, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.186959482 (16h13m53.970299571s) [noise=true , address=212.102.52.215:4384 , piece_id=R5RDE456F34SAVPUBICPDJGAXHYOOH4ILTDIBAKLGL34PMMFVYQA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.188117592 (16h13m53.971457691s) [noise=true , address=212.102.52.214:4359 , piece_id=QDTWWS3LMTJONF3G2GZE3OLYUCXUNMKAOWF3GCZXQHXEE2XWY5YQ, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.193324387 (16h13m53.976664486s) [noise=true , address=212.102.52.215:4376 , piece_id=MASYN2GTW4OGTF422XYQL2K4JIV633OFCQAT32ZHHBPUXDRZAHFA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.194251486 (16h13m53.977591575s) [noise=true , address=212.102.52.213:4342 , piece_id=WNWW2PC5W7VYO5ZPOIY4GFUUEL2BNUDEBGMX53KXQQTRQAN744TA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.194415629 (16h13m53.977755728s) [noise=true , address=212.102.52.213:4337 , piece_id=VWF4LCALY4RIE3KAFYNGANGUCHOCB7G7WS5IM2S2TXTT22ZCIICQ, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.196030210 (16h13m53.979370309s) [noise=true , address=212.102.52.214:4368 , piece_id=DFPNMKRLFN43RCYCIO5TMOSIVWLMBJPNPZMO5BZ3AEKDC5SARLQA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.198914643 (16h13m53.982254742s) [noise=true , address=212.102.52.214:4361 , piece_id=M53YWUBS6EJZK4IZWS6P7T3EJ7CPWXYL3CH7TRZW7CJZHQCUPXGA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.200106938 (16h13m53.983447037s) [noise=true , address=212.102.52.215:4381 , piece_id=WAPMAH3XPQPJVQUOZBNONWMFCT7T3LGFX7BMTHMAPU4PIZAMQ2MA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.201764692 (16h13m53.985104791s) [noise=true , address=212.102.52.215:4389 , piece_id=FDYCELU27TEDNDALWQ565K3AE5FVUMGLOIQ3PUIIVIXUAMYPUKHA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.203016379 (16h13m53.986356478s) [noise=true , address=212.102.52.214:4364 , piece_id=IISEDBNGQ7EW5BWECJ5M7E56YP5XAA5VP3JEMYFPMRYHE2KN3HXQ, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.208385454 (16h13m53.991725543s) [noise=true , address=212.102.52.216:4415 , piece_id=NFEG3MWKNN3YBCXLTJYOMILKA4HSCSIIQIBYV5CFNNNG4HR2C6XA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.209472508 (16h13m53.992812597s) [noise=true , address=212.102.52.216:4395 , piece_id=MY2C5EYWIBPIGG5OGRA4NAYL7QYNIZVBWT5ZBRF2HHBC2EGZY5EQ, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.220860086 (16h13m54.004200185s) [noise=true , address=212.102.52.217:4437 , piece_id=WGOJRDIPH35CPTDVN2TZ5IBN3RY4DBVMLA2WPFNWMEVDYDVKEJ2Q, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.221098722 (16h13m54.004438821s) [noise=true , address=212.102.52.217:4439 , piece_id=TOHCWZLYIUS7QE4N4ZAIB2QWUAVB2DE2SAZTG6OMQLWR5VVDDRNA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.221131835 (16h13m54.004471934s) [seg_pos={0 0}, upload=15534]: Satisfied cohort requirements after 16 pieces
May 25 18:18:08.221311708 (16h13m54.004651807s) [noise=true , address=212.102.52.217:4432 , piece_id=UF3ANPTDI6H3Z4IG3HQIDLCA4J32MKTXAND6XYJ2HNT22OKMGDHQ, seg_pos={0 0}, upload=15534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CPW6Lyzfry6DmU86Fh3rW6u1RS3qpe79mYMaJRB5r4QEEQC6F): protocol: expected piece hash; context canceled; EOF
May 25 18:18:08.221321857 (16h13m54.004661956s) [noise=true , address=212.102.52.217:4432 , piece_id=UF3ANPTDI6H3Z4IG3HQIDLCA4J32MKTXAND6XYJ2HNT22OKMGDHQ, seg_pos={0 0}, upload=15534]: Long tail cancellation due to optimal threshold
May 25 18:18:08.221333039 (16h13m54.004673138s) [noise=true , address=212.102.52.216:4410 , piece_id=CNVT54RWTRYXAKSZW7SXKAFTJXJ7TJOMNMQV4SEZAWX6P25SMDAA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BiVenoRJiFZVtT6vhvPQLfz3pScWYJ1r2X9CLRg2F2jc8MALC): protocol: expected piece hash; context canceled; EOF
May 25 18:18:08.221338569 (16h13m54.004678658s) [noise=true , address=212.102.52.216:4410 , piece_id=CNVT54RWTRYXAKSZW7SXKAFTJXJ7TJOMNMQV4SEZAWX6P25SMDAA, seg_pos={0 0}, upload=15534]: Long tail cancellation due to optimal threshold
May 25 18:18:08.221446795 (16h13m54.004786884s) [noise=true , address=212.102.52.213:4331 , piece_id=EEG5YWPYZRZ5A55GPCGE7ZJCA3EOMYF6MJVKFOWTMDYPAZD7P2WQ, seg_pos={0 0}, upload=15534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eyfVMwFVb4Q5nbc54PSLebuyWfm2rCgCfGrNc6AovSWqe6Upn): protocol: expected piece hash; context canceled; EOF
May 25 18:18:08.221453309 (16h13m54.004793408s) [noise=true , address=212.102.52.213:4331 , piece_id=EEG5YWPYZRZ5A55GPCGE7ZJCA3EOMYF6MJVKFOWTMDYPAZD7P2WQ, seg_pos={0 0}, upload=15534]: Long tail cancellation due to optimal threshold
May 25 18:18:08.221554672 (16h13m54.004894771s) [noise=true , address=212.102.52.216:4416 , piece_id=KRVXWG4J4O4S7NJK2ZM7RYQQBXHGV3TEPDRCHAOSQZ7PDPBYSWZQ, seg_pos={0 0}, upload=15534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 18:18:08.221560422 (16h13m54.004900521s) [noise=true , address=212.102.52.216:4416 , piece_id=KRVXWG4J4O4S7NJK2ZM7RYQQBXHGV3TEPDRCHAOSQZ7PDPBYSWZQ, seg_pos={0 0}, upload=15534]: Long tail cancellation due to optimal threshold
May 25 18:18:08.221584258 (16h13m54.004924357s) [seg_pos={0 0}, upload=15534]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 18:18:08.221659130 (16h13m54.004999229s) [upload=15534]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 18:18:08.221664431 (16h13m54.00500453s) [upload=15534]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 18:18:23.826937276 (16h14m9.610277375s) [upload=15534]: Done uploading segments...
May 25 18:18:23.826954218 (16h14m9.610294317s) [upload=15534]: Upload finished. err: <nil>
Duration: 29.83065156s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...............................X............................] duration:
3.714020677s trace:
https://jaeger.dev.storj.io/trace/14eb8131f680ed75
FAIL (2026-05-25 18:19:41 UTC)
orbiter warning: context deadline exceeded
Duration: 3.71404331s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
13 out of 60 is failed (threshold=3) [......X....X.......X......X..XX.XX.X....X.....XX......X.....] duration:
6.59683354s trace:
https://jaeger.dev.storj.io/trace/698dfada1f99867a
FAIL (2026-05-25 18:19:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.596852756s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
13 out of 60 is failed (threshold=3) [.......X....X.......X......X..XX.XX.X....X.....XX......X....] duration:
41.057346869s trace:
https://jaeger.dev.storj.io/trace/698dfada1f99867a
FAIL (2026-05-25 18:19:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 41.057359293s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [................................X...........................] duration:
3.829195082s trace:
https://jaeger.dev.storj.io/trace/14eb8131f680ed75
FAIL (2026-05-25 18:19:18 UTC)
orbiter warning: context deadline exceeded
Duration: 3.829205663s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
680.633646ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/7e710747942acdb2
FAIL (2026-05-25 18:19:00 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 680.656712ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
3 out of 60 is failed (threshold=3) [.........................X................X...............X.] duration:
3.967780743s trace:
https://jaeger.dev.storj.io/trace/47c8961ca375d003
FAIL (2026-05-25 18:18:39 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
DVKEJ2Q, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.826862166 (16h13m52.610202265s) [noise=true , address=212.102.52.216:4410 , piece_id=CNVT54RWTRYXAKSZW7SXKAFTJXJ7TJOMNMQV4SEZAWX6P25SMDAA, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.827074101 (16h13m52.6104142s) [noise=true , address=212.102.52.214:4359 , piece_id=QDTWWS3LMTJONF3G2GZE3OLYUCXUNMKAOWF3GCZXQHXEE2XWY5YQ, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.827499863 (16h13m52.610839962s) [noise=true , address=212.102.52.213:4331 , piece_id=EEG5YWPYZRZ5A55GPCGE7ZJCA3EOMYF6MJVKFOWTMDYPAZD7P2WQ, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.827663575 (16h13m52.611003674s) [noise=true , address=212.102.52.215:4376 , piece_id=MASYN2GTW4OGTF422XYQL2K4JIV633OFCQAT32ZHHBPUXDRZAHFA, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.827904234 (16h13m52.611244333s) [noise=true , address=212.102.52.217:4439 , piece_id=TOHCWZLYIUS7QE4N4ZAIB2QWUAVB2DE2SAZTG6OMQLWR5VVDDRNA, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.828316992 (16h13m52.611657091s) [noise=true , address=212.102.52.213:4323 , piece_id=VKBPFLZMPGTGYIRKDJ7CYJVIGRSBZ7BL4BRRYAPFOCXJZF3DHEGQ, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.828439305 (16h13m52.611779394s) [noise=true , address=212.102.52.213:4337 , piece_id=VWF4LCALY4RIE3KAFYNGANGUCHOCB7G7WS5IM2S2TXTT22ZCIICQ, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.828885316 (16h13m52.612225416s) [noise=true , address=212.102.52.213:4342 , piece_id=WNWW2PC5W7VYO5ZPOIY4GFUUEL2BNUDEBGMX53KXQQTRQAN744TA, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.829013611 (16h13m52.61235371s) [noise=true , address=212.102.52.214:4361 , piece_id=M53YWUBS6EJZK4IZWS6P7T3EJ7CPWXYL3CH7TRZW7CJZHQCUPXGA, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.829318423 (16h13m52.612658522s) [noise=true , address=212.102.52.214:4368 , piece_id=DFPNMKRLFN43RCYCIO5TMOSIVWLMBJPNPZMO5BZ3AEKDC5SARLQA, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.829676847 (16h13m52.613016946s) [noise=true , address=212.102.52.217:4432 , piece_id=UF3ANPTDI6H3Z4IG3HQIDLCA4J32MKTXAND6XYJ2HNT22OKMGDHQ, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.829790153 (16h13m52.613130252s) [noise=true , address=212.102.52.215:4389 , piece_id=FDYCELU27TEDNDALWQ565K3AE5FVUMGLOIQ3PUIIVIXUAMYPUKHA, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.830404054 (16h13m52.613744153s) [noise=true , address=212.102.52.214:4364 , piece_id=IISEDBNGQ7EW5BWECJ5M7E56YP5XAA5VP3JEMYFPMRYHE2KN3HXQ, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.831055047 (16h13m52.614395146s) [noise=true , address=212.102.52.215:4381 , piece_id=WAPMAH3XPQPJVQUOZBNONWMFCT7T3LGFX7BMTHMAPU4PIZAMQ2MA, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.831772195 (16h13m52.615112304s) [noise=true , address=212.102.52.215:4384 , piece_id=R5RDE456F34SAVPUBICPDJGAXHYOOH4ILTDIBAKLGL34PMMFVYQA, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:07.900796750 (16h13m53.68413685s) [upload=15534]: Next returned nil segment
May 25 18:18:07.900813633 (16h13m53.684153722s) [upload=15534]: Waiting for error group managing segments...
May 25 18:18:08.180403392 (16h13m53.963743492s) [noise=true , address=212.102.52.213:4323 , piece_id=VKBPFLZMPGTGYIRKDJ7CYJVIGRSBZ7BL4BRRYAPFOCXJZF3DHEGQ, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.185311568 (16h13m53.968651657s) [noise=true , address=212.102.52.217:4419 , piece_id=JYS43OLSNC3HGNDCICQHGZ4NV5PDKSP4PZGVQYGT7LFJXDVLDFEQ, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.186959482 (16h13m53.970299571s) [noise=true , address=212.102.52.215:4384 , piece_id=R5RDE456F34SAVPUBICPDJGAXHYOOH4ILTDIBAKLGL34PMMFVYQA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.188117592 (16h13m53.971457691s) [noise=true , address=212.102.52.214:4359 , piece_id=QDTWWS3LMTJONF3G2GZE3OLYUCXUNMKAOWF3GCZXQHXEE2XWY5YQ, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.193324387 (16h13m53.976664486s) [noise=true , address=212.102.52.215:4376 , piece_id=MASYN2GTW4OGTF422XYQL2K4JIV633OFCQAT32ZHHBPUXDRZAHFA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.194251486 (16h13m53.977591575s) [noise=true , address=212.102.52.213:4342 , piece_id=WNWW2PC5W7VYO5ZPOIY4GFUUEL2BNUDEBGMX53KXQQTRQAN744TA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.194415629 (16h13m53.977755728s) [noise=true , address=212.102.52.213:4337 , piece_id=VWF4LCALY4RIE3KAFYNGANGUCHOCB7G7WS5IM2S2TXTT22ZCIICQ, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.196030210 (16h13m53.979370309s) [noise=true , address=212.102.52.214:4368 , piece_id=DFPNMKRLFN43RCYCIO5TMOSIVWLMBJPNPZMO5BZ3AEKDC5SARLQA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.198914643 (16h13m53.982254742s) [noise=true , address=212.102.52.214:4361 , piece_id=M53YWUBS6EJZK4IZWS6P7T3EJ7CPWXYL3CH7TRZW7CJZHQCUPXGA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.200106938 (16h13m53.983447037s) [noise=true , address=212.102.52.215:4381 , piece_id=WAPMAH3XPQPJVQUOZBNONWMFCT7T3LGFX7BMTHMAPU4PIZAMQ2MA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.201764692 (16h13m53.985104791s) [noise=true , address=212.102.52.215:4389 , piece_id=FDYCELU27TEDNDALWQ565K3AE5FVUMGLOIQ3PUIIVIXUAMYPUKHA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.203016379 (16h13m53.986356478s) [noise=true , address=212.102.52.214:4364 , piece_id=IISEDBNGQ7EW5BWECJ5M7E56YP5XAA5VP3JEMYFPMRYHE2KN3HXQ, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.208385454 (16h13m53.991725543s) [noise=true , address=212.102.52.216:4415 , piece_id=NFEG3MWKNN3YBCXLTJYOMILKA4HSCSIIQIBYV5CFNNNG4HR2C6XA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.209472508 (16h13m53.992812597s) [noise=true , address=212.102.52.216:4395 , piece_id=MY2C5EYWIBPIGG5OGRA4NAYL7QYNIZVBWT5ZBRF2HHBC2EGZY5EQ, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.220860086 (16h13m54.004200185s) [noise=true , address=212.102.52.217:4437 , piece_id=WGOJRDIPH35CPTDVN2TZ5IBN3RY4DBVMLA2WPFNWMEVDYDVKEJ2Q, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.221098722 (16h13m54.004438821s) [noise=true , address=212.102.52.217:4439 , piece_id=TOHCWZLYIUS7QE4N4ZAIB2QWUAVB2DE2SAZTG6OMQLWR5VVDDRNA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.221131835 (16h13m54.004471934s) [seg_pos={0 0}, upload=15534]: Satisfied cohort requirements after 16 pieces
May 25 18:18:08.221311708 (16h13m54.004651807s) [noise=true , address=212.102.52.217:4432 , piece_id=UF3ANPTDI6H3Z4IG3HQIDLCA4J32MKTXAND6XYJ2HNT22OKMGDHQ, seg_pos={0 0}, upload=15534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CPW6Lyzfry6DmU86Fh3rW6u1RS3qpe79mYMaJRB5r4QEEQC6F): protocol: expected piece hash; context canceled; EOF
May 25 18:18:08.221321857 (16h13m54.004661956s) [noise=true , address=212.102.52.217:4432 , piece_id=UF3ANPTDI6H3Z4IG3HQIDLCA4J32MKTXAND6XYJ2HNT22OKMGDHQ, seg_pos={0 0}, upload=15534]: Long tail cancellation due to optimal threshold
May 25 18:18:08.221333039 (16h13m54.004673138s) [noise=true , address=212.102.52.216:4410 , piece_id=CNVT54RWTRYXAKSZW7SXKAFTJXJ7TJOMNMQV4SEZAWX6P25SMDAA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BiVenoRJiFZVtT6vhvPQLfz3pScWYJ1r2X9CLRg2F2jc8MALC): protocol: expected piece hash; context canceled; EOF
May 25 18:18:08.221338569 (16h13m54.004678658s) [noise=true , address=212.102.52.216:4410 , piece_id=CNVT54RWTRYXAKSZW7SXKAFTJXJ7TJOMNMQV4SEZAWX6P25SMDAA, seg_pos={0 0}, upload=15534]: Long tail cancellation due to optimal threshold
May 25 18:18:08.221446795 (16h13m54.004786884s) [noise=true , address=212.102.52.213:4331 , piece_id=EEG5YWPYZRZ5A55GPCGE7ZJCA3EOMYF6MJVKFOWTMDYPAZD7P2WQ, seg_pos={0 0}, upload=15534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eyfVMwFVb4Q5nbc54PSLebuyWfm2rCgCfGrNc6AovSWqe6Upn): protocol: expected piece hash; context canceled; EOF
May 25 18:18:08.221453309 (16h13m54.004793408s) [noise=true , address=212.102.52.213:4331 , piece_id=EEG5YWPYZRZ5A55GPCGE7ZJCA3EOMYF6MJVKFOWTMDYPAZD7P2WQ, seg_pos={0 0}, upload=15534]: Long tail cancellation due to optimal threshold
May 25 18:18:08.221554672 (16h13m54.004894771s) [noise=true , address=212.102.52.216:4416 , piece_id=KRVXWG4J4O4S7NJK2ZM7RYQQBXHGV3TEPDRCHAOSQZ7PDPBYSWZQ, seg_pos={0 0}, upload=15534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 18:18:08.221560422 (16h13m54.004900521s) [noise=true , address=212.102.52.216:4416 , piece_id=KRVXWG4J4O4S7NJK2ZM7RYQQBXHGV3TEPDRCHAOSQZ7PDPBYSWZQ, seg_pos={0 0}, upload=15534]: Long tail cancellation due to optimal threshold
May 25 18:18:08.221584258 (16h13m54.004924357s) [seg_pos={0 0}, upload=15534]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 18:18:08.221659130 (16h13m54.004999229s) [upload=15534]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 18:18:08.221664431 (16h13m54.00500453s) [upload=15534]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 18:18:23.826937276 (16h14m9.610277375s) [upload=15534]: Done uploading segments...
May 25 18:18:23.826954218 (16h14m9.610294317s) [upload=15534]: Upload finished. err: <nil>
Duration: 3.967792175s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
3 out of 60 is failed (threshold=3) [..........................X................X...............X] duration:
1m0.002850381s trace:
https://jaeger.dev.storj.io/trace/47c8961ca375d003
FAIL (2026-05-25 18:18:35 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
DVKEJ2Q, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.826862166 (16h13m52.610202265s) [noise=true , address=212.102.52.216:4410 , piece_id=CNVT54RWTRYXAKSZW7SXKAFTJXJ7TJOMNMQV4SEZAWX6P25SMDAA, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.827074101 (16h13m52.6104142s) [noise=true , address=212.102.52.214:4359 , piece_id=QDTWWS3LMTJONF3G2GZE3OLYUCXUNMKAOWF3GCZXQHXEE2XWY5YQ, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.827499863 (16h13m52.610839962s) [noise=true , address=212.102.52.213:4331 , piece_id=EEG5YWPYZRZ5A55GPCGE7ZJCA3EOMYF6MJVKFOWTMDYPAZD7P2WQ, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.827663575 (16h13m52.611003674s) [noise=true , address=212.102.52.215:4376 , piece_id=MASYN2GTW4OGTF422XYQL2K4JIV633OFCQAT32ZHHBPUXDRZAHFA, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.827904234 (16h13m52.611244333s) [noise=true , address=212.102.52.217:4439 , piece_id=TOHCWZLYIUS7QE4N4ZAIB2QWUAVB2DE2SAZTG6OMQLWR5VVDDRNA, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.828316992 (16h13m52.611657091s) [noise=true , address=212.102.52.213:4323 , piece_id=VKBPFLZMPGTGYIRKDJ7CYJVIGRSBZ7BL4BRRYAPFOCXJZF3DHEGQ, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.828439305 (16h13m52.611779394s) [noise=true , address=212.102.52.213:4337 , piece_id=VWF4LCALY4RIE3KAFYNGANGUCHOCB7G7WS5IM2S2TXTT22ZCIICQ, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.828885316 (16h13m52.612225416s) [noise=true , address=212.102.52.213:4342 , piece_id=WNWW2PC5W7VYO5ZPOIY4GFUUEL2BNUDEBGMX53KXQQTRQAN744TA, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.829013611 (16h13m52.61235371s) [noise=true , address=212.102.52.214:4361 , piece_id=M53YWUBS6EJZK4IZWS6P7T3EJ7CPWXYL3CH7TRZW7CJZHQCUPXGA, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.829318423 (16h13m52.612658522s) [noise=true , address=212.102.52.214:4368 , piece_id=DFPNMKRLFN43RCYCIO5TMOSIVWLMBJPNPZMO5BZ3AEKDC5SARLQA, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.829676847 (16h13m52.613016946s) [noise=true , address=212.102.52.217:4432 , piece_id=UF3ANPTDI6H3Z4IG3HQIDLCA4J32MKTXAND6XYJ2HNT22OKMGDHQ, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.829790153 (16h13m52.613130252s) [noise=true , address=212.102.52.215:4389 , piece_id=FDYCELU27TEDNDALWQ565K3AE5FVUMGLOIQ3PUIIVIXUAMYPUKHA, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.830404054 (16h13m52.613744153s) [noise=true , address=212.102.52.214:4364 , piece_id=IISEDBNGQ7EW5BWECJ5M7E56YP5XAA5VP3JEMYFPMRYHE2KN3HXQ, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.831055047 (16h13m52.614395146s) [noise=true , address=212.102.52.215:4381 , piece_id=WAPMAH3XPQPJVQUOZBNONWMFCT7T3LGFX7BMTHMAPU4PIZAMQ2MA, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:06.831772195 (16h13m52.615112304s) [noise=true , address=212.102.52.215:4384 , piece_id=R5RDE456F34SAVPUBICPDJGAXHYOOH4ILTDIBAKLGL34PMMFVYQA, seg_pos={0 0}, upload=15534]: Uploading piece...
May 25 18:18:07.900796750 (16h13m53.68413685s) [upload=15534]: Next returned nil segment
May 25 18:18:07.900813633 (16h13m53.684153722s) [upload=15534]: Waiting for error group managing segments...
May 25 18:18:08.180403392 (16h13m53.963743492s) [noise=true , address=212.102.52.213:4323 , piece_id=VKBPFLZMPGTGYIRKDJ7CYJVIGRSBZ7BL4BRRYAPFOCXJZF3DHEGQ, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.185311568 (16h13m53.968651657s) [noise=true , address=212.102.52.217:4419 , piece_id=JYS43OLSNC3HGNDCICQHGZ4NV5PDKSP4PZGVQYGT7LFJXDVLDFEQ, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.186959482 (16h13m53.970299571s) [noise=true , address=212.102.52.215:4384 , piece_id=R5RDE456F34SAVPUBICPDJGAXHYOOH4ILTDIBAKLGL34PMMFVYQA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.188117592 (16h13m53.971457691s) [noise=true , address=212.102.52.214:4359 , piece_id=QDTWWS3LMTJONF3G2GZE3OLYUCXUNMKAOWF3GCZXQHXEE2XWY5YQ, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.193324387 (16h13m53.976664486s) [noise=true , address=212.102.52.215:4376 , piece_id=MASYN2GTW4OGTF422XYQL2K4JIV633OFCQAT32ZHHBPUXDRZAHFA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.194251486 (16h13m53.977591575s) [noise=true , address=212.102.52.213:4342 , piece_id=WNWW2PC5W7VYO5ZPOIY4GFUUEL2BNUDEBGMX53KXQQTRQAN744TA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.194415629 (16h13m53.977755728s) [noise=true , address=212.102.52.213:4337 , piece_id=VWF4LCALY4RIE3KAFYNGANGUCHOCB7G7WS5IM2S2TXTT22ZCIICQ, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.196030210 (16h13m53.979370309s) [noise=true , address=212.102.52.214:4368 , piece_id=DFPNMKRLFN43RCYCIO5TMOSIVWLMBJPNPZMO5BZ3AEKDC5SARLQA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.198914643 (16h13m53.982254742s) [noise=true , address=212.102.52.214:4361 , piece_id=M53YWUBS6EJZK4IZWS6P7T3EJ7CPWXYL3CH7TRZW7CJZHQCUPXGA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.200106938 (16h13m53.983447037s) [noise=true , address=212.102.52.215:4381 , piece_id=WAPMAH3XPQPJVQUOZBNONWMFCT7T3LGFX7BMTHMAPU4PIZAMQ2MA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.201764692 (16h13m53.985104791s) [noise=true , address=212.102.52.215:4389 , piece_id=FDYCELU27TEDNDALWQ565K3AE5FVUMGLOIQ3PUIIVIXUAMYPUKHA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.203016379 (16h13m53.986356478s) [noise=true , address=212.102.52.214:4364 , piece_id=IISEDBNGQ7EW5BWECJ5M7E56YP5XAA5VP3JEMYFPMRYHE2KN3HXQ, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.208385454 (16h13m53.991725543s) [noise=true , address=212.102.52.216:4415 , piece_id=NFEG3MWKNN3YBCXLTJYOMILKA4HSCSIIQIBYV5CFNNNG4HR2C6XA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.209472508 (16h13m53.992812597s) [noise=true , address=212.102.52.216:4395 , piece_id=MY2C5EYWIBPIGG5OGRA4NAYL7QYNIZVBWT5ZBRF2HHBC2EGZY5EQ, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.220860086 (16h13m54.004200185s) [noise=true , address=212.102.52.217:4437 , piece_id=WGOJRDIPH35CPTDVN2TZ5IBN3RY4DBVMLA2WPFNWMEVDYDVKEJ2Q, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.221098722 (16h13m54.004438821s) [noise=true , address=212.102.52.217:4439 , piece_id=TOHCWZLYIUS7QE4N4ZAIB2QWUAVB2DE2SAZTG6OMQLWR5VVDDRNA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: <nil>
May 25 18:18:08.221131835 (16h13m54.004471934s) [seg_pos={0 0}, upload=15534]: Satisfied cohort requirements after 16 pieces
May 25 18:18:08.221311708 (16h13m54.004651807s) [noise=true , address=212.102.52.217:4432 , piece_id=UF3ANPTDI6H3Z4IG3HQIDLCA4J32MKTXAND6XYJ2HNT22OKMGDHQ, seg_pos={0 0}, upload=15534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CPW6Lyzfry6DmU86Fh3rW6u1RS3qpe79mYMaJRB5r4QEEQC6F): protocol: expected piece hash; context canceled; EOF
May 25 18:18:08.221321857 (16h13m54.004661956s) [noise=true , address=212.102.52.217:4432 , piece_id=UF3ANPTDI6H3Z4IG3HQIDLCA4J32MKTXAND6XYJ2HNT22OKMGDHQ, seg_pos={0 0}, upload=15534]: Long tail cancellation due to optimal threshold
May 25 18:18:08.221333039 (16h13m54.004673138s) [noise=true , address=212.102.52.216:4410 , piece_id=CNVT54RWTRYXAKSZW7SXKAFTJXJ7TJOMNMQV4SEZAWX6P25SMDAA, seg_pos={0 0}, upload=15534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BiVenoRJiFZVtT6vhvPQLfz3pScWYJ1r2X9CLRg2F2jc8MALC): protocol: expected piece hash; context canceled; EOF
May 25 18:18:08.221338569 (16h13m54.004678658s) [noise=true , address=212.102.52.216:4410 , piece_id=CNVT54RWTRYXAKSZW7SXKAFTJXJ7TJOMNMQV4SEZAWX6P25SMDAA, seg_pos={0 0}, upload=15534]: Long tail cancellation due to optimal threshold
May 25 18:18:08.221446795 (16h13m54.004786884s) [noise=true , address=212.102.52.213:4331 , piece_id=EEG5YWPYZRZ5A55GPCGE7ZJCA3EOMYF6MJVKFOWTMDYPAZD7P2WQ, seg_pos={0 0}, upload=15534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eyfVMwFVb4Q5nbc54PSLebuyWfm2rCgCfGrNc6AovSWqe6Upn): protocol: expected piece hash; context canceled; EOF
May 25 18:18:08.221453309 (16h13m54.004793408s) [noise=true , address=212.102.52.213:4331 , piece_id=EEG5YWPYZRZ5A55GPCGE7ZJCA3EOMYF6MJVKFOWTMDYPAZD7P2WQ, seg_pos={0 0}, upload=15534]: Long tail cancellation due to optimal threshold
May 25 18:18:08.221554672 (16h13m54.004894771s) [noise=true , address=212.102.52.216:4416 , piece_id=KRVXWG4J4O4S7NJK2ZM7RYQQBXHGV3TEPDRCHAOSQZ7PDPBYSWZQ, seg_pos={0 0}, upload=15534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 18:18:08.221560422 (16h13m54.004900521s) [noise=true , address=212.102.52.216:4416 , piece_id=KRVXWG4J4O4S7NJK2ZM7RYQQBXHGV3TEPDRCHAOSQZ7PDPBYSWZQ, seg_pos={0 0}, upload=15534]: Long tail cancellation due to optimal threshold
May 25 18:18:08.221584258 (16h13m54.004924357s) [seg_pos={0 0}, upload=15534]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 18:18:08.221659130 (16h13m54.004999229s) [upload=15534]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 18:18:08.221664431 (16h13m54.00500453s) [upload=15534]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 18:18:23.826937276 (16h14m9.610277375s) [upload=15534]: Done uploading segments...
May 25 18:18:23.826954218 (16h14m9.610294317s) [upload=15534]: Upload finished. err: <nil>
Duration: 1m0.002867293s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.................................X..........................] duration:
3.730204727s trace:
https://jaeger.dev.storj.io/trace/14eb8131f680ed75
FAIL (2026-05-25 18:18:24 UTC)
orbiter warning: context deadline exceeded
Duration: 3.730223642s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..................................X.........................] duration:
3.759056957s trace:
https://jaeger.dev.storj.io/trace/14eb8131f680ed75
FAIL (2026-05-25 18:18:03 UTC)
orbiter warning: context deadline exceeded
Duration: 3.759091574s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
13 out of 60 is failed (threshold=3) [........X....X.......X......X..XX.XX.X....X.....XX......X...] duration:
6.718151342s trace:
https://jaeger.dev.storj.io/trace/698dfada1f99867a
FAIL (2026-05-25 18:17:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.718176839s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
678.156142ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/7e710747942acdb2
FAIL (2026-05-25 18:17:50 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 678.179857ms
core
satellite
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...................................X........................] duration:
4.733737981s trace:
https://jaeger.dev.storj.io/trace/14eb8131f680ed75
FAIL (2026-05-25 18:17:19 UTC)
orbiter warning: context deadline exceeded
Duration: 4.733759962s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [...........................X................X...............] duration:
3.919348339s trace:
https://jaeger.dev.storj.io/trace/467f79c1decb330
FAIL (2026-05-25 18:17:12 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
se=true , address=212.102.52.216:4416 , piece_id=AMAJVMUOCVS5TOZLHMDOUSQ5YXCFSYEDT7BW4CW56M6VA6LIDJLA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.813733143 (16h3m2.597073242s) [noise=true , address=212.102.52.213:4340 , piece_id=GGQZRBZNC5DKXOZYV664XOFD2YUAZAL4RPRDQY7ZY6LUN6NKEFKQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.814170728 (16h3m2.597510827s) [noise=true , address=212.102.52.214:4356 , piece_id=5OQLNMQN2NWE2OJO7FS44O6NI4NRMDCGYQQ3OTTH62CQDZZGHVQA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.814532990 (16h3m2.597873089s) [noise=true , address=212.102.52.216:4405 , piece_id=CBCNKWI2VLVCUZDVM54E243ULDQUGYEH5EWRK2ES6V6LRUPWQIIA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.814792664 (16h3m2.598132764s) [noise=true , address=212.102.52.214:4358 , piece_id=NDE6FEYH2OTGRTRIHGJBGJSV77ZUKHXAR4AON3NDKBU33EL4BGZQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.815026731 (16h3m2.59836683s) [noise=true , address=212.102.52.214:4347 , piece_id=6SXGEUYVLSOUSGPUY22UAXQKUWFMWVQOV3NHMMOOJY3NNL5XBAHA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.815255207 (16h3m2.598595296s) [noise=true , address=212.102.52.215:4369 , piece_id=TLNVSD3KWUSXOW3YLEMNRREQW2RMZABFJEEGX64XH7QYAXZBNDHQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.815526655 (16h3m2.598866754s) [noise=true , address=212.102.52.213:4343 , piece_id=JA6JX5GQOL6WT6MQV4G4UZ2M73QAW5ZII5OW5EO367VKYRESOGGA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.817053338 (16h3m2.600393437s) [noise=true , address=212.102.52.213:4328 , piece_id=GUZPGD4HCE5PSNWBWZMLEDHA5AEMZZTEF7N2V4OXC4LCI7TIRIFQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.817184408 (16h3m2.600524507s) [noise=true , address=212.102.52.215:4378 , piece_id=6456XMZDE4RFGHOTOUDM2SNQJDOCHHERTJKKQA6R6KU4EQCD74XA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.818090236 (16h3m2.601430335s) [noise=true , address=212.102.52.216:4408 , piece_id=IEQIPNTTQ3LN7LVVVQSSQU67NLWWG5DC6D5I5W33SKPKPN532DMQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.818269799 (16h3m2.601609898s) [noise=true , address=212.102.52.217:4418 , piece_id=EVBLYS5IQFKBL5TE3UTNM5SENH3COOX6W4JDF53SFFON3JOYQHMQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.819170618 (16h3m2.602510717s) [noise=true , address=212.102.52.217:4432 , piece_id=GZKMD3YNEMZNXRNHAPFBBXL6T7XOFUHKVJURDCJQ6CTCZRDUW4KQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.819559901 (16h3m2.6029s) [noise=true , address=212.102.52.217:4426 , piece_id=CK4UNDBYQQCEVPTT5WGWHBWFH6CRZO3JO7RZL3RO5NOYDFX3DJNA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.820329489 (16h3m2.603669588s) [noise=true , address=212.102.52.217:4431 , piece_id=3JXJF2YLDKTQDXDK666LSTYGYTMHSE3ZXJ7Y5WOUHAXWZVFRBRYA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.821320470 (16h3m2.604660559s) [noise=true , address=212.102.52.215:4388 , piece_id=DNXSHYL6IAMUWAUUVCH6WANAI3HSQIMVJYKSSG3YKLKNS4S6DGJQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:17.875060315 (16h3m3.658400414s) [upload=15359]: Next returned nil segment
May 25 18:07:17.875071437 (16h3m3.658411536s) [upload=15359]: Waiting for error group managing segments...
May 25 18:07:18.161378706 (16h3m3.944718805s) [noise=true , address=212.102.52.214:4356 , piece_id=5OQLNMQN2NWE2OJO7FS44O6NI4NRMDCGYQQ3OTTH62CQDZZGHVQA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.163883035 (16h3m3.947223134s) [noise=true , address=212.102.52.215:4369 , piece_id=TLNVSD3KWUSXOW3YLEMNRREQW2RMZABFJEEGX64XH7QYAXZBNDHQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.169179691 (16h3m3.9525198s) [noise=true , address=212.102.52.214:4347 , piece_id=6SXGEUYVLSOUSGPUY22UAXQKUWFMWVQOV3NHMMOOJY3NNL5XBAHA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.169953397 (16h3m3.953293496s) [noise=true , address=212.102.52.214:4358 , piece_id=NDE6FEYH2OTGRTRIHGJBGJSV77ZUKHXAR4AON3NDKBU33EL4BGZQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.170012670 (16h3m3.953352769s) [noise=true , address=212.102.52.216:4406 , piece_id=DQK2NNDQZWMM7KWBUMYVDJZTGT3SITN3M2ILQ7Y3QNIBV4WZXTTQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.173457273 (16h3m3.956797402s) [noise=true , address=212.102.52.217:4431 , piece_id=3JXJF2YLDKTQDXDK666LSTYGYTMHSE3ZXJ7Y5WOUHAXWZVFRBRYA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.173853839 (16h3m3.957193938s) [noise=true , address=212.102.52.213:4340 , piece_id=GGQZRBZNC5DKXOZYV664XOFD2YUAZAL4RPRDQY7ZY6LUN6NKEFKQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.175372487 (16h3m3.958712586s) [noise=true , address=212.102.52.216:4405 , piece_id=CBCNKWI2VLVCUZDVM54E243ULDQUGYEH5EWRK2ES6V6LRUPWQIIA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.181671326 (16h3m3.965011425s) [noise=true , address=212.102.52.215:4376 , piece_id=RK5KEGAOHFZBU4NN7IO44XPBY5YAOANJ5VLPUU3XVBIGVVFSFZVA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.186238400 (16h3m3.969578509s) [noise=true , address=212.102.52.217:4418 , piece_id=EVBLYS5IQFKBL5TE3UTNM5SENH3COOX6W4JDF53SFFON3JOYQHMQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.190371015 (16h3m3.973711114s) [noise=true , address=212.102.52.213:4328 , piece_id=GUZPGD4HCE5PSNWBWZMLEDHA5AEMZZTEF7N2V4OXC4LCI7TIRIFQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.191589199 (16h3m3.974929298s) [noise=true , address=212.102.52.216:4408 , piece_id=IEQIPNTTQ3LN7LVVVQSSQU67NLWWG5DC6D5I5W33SKPKPN532DMQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.199054834 (16h3m3.982394933s) [noise=true , address=212.102.52.215:4388 , piece_id=DNXSHYL6IAMUWAUUVCH6WANAI3HSQIMVJYKSSG3YKLKNS4S6DGJQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.220275774 (16h3m4.003615873s) [noise=true , address=212.102.52.217:4426 , piece_id=CK4UNDBYQQCEVPTT5WGWHBWFH6CRZO3JO7RZL3RO5NOYDFX3DJNA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.220628726 (16h3m4.003968825s) [noise=true , address=212.102.52.215:4378 , piece_id=6456XMZDE4RFGHOTOUDM2SNQJDOCHHERTJKKQA6R6KU4EQCD74XA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.226546558 (16h3m4.009886657s) [noise=true , address=212.102.52.213:4321 , piece_id=QFBCH5MLCAIFPWMHO4NNQJLPIX7QG5LPQJUHG77WMPI6VPIBQIQA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.226564292 (16h3m4.009904421s) [seg_pos={0 0}, upload=15359]: Satisfied cohort requirements after 16 pieces
May 25 18:07:18.226741169 (16h3m4.010081268s) [noise=true , address=212.102.52.214:4368 , piece_id=OPJWV4TDPYNLLY2A5TSKGCUGX542YUX2OMPMILUTPE3TJR4L72AA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zavf82Jv8HWUWhuGf2nXnhiF1GDupLBVhu6urZQP6N6j71R3C): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.226748514 (16h3m4.010088613s) [noise=true , address=212.102.52.214:4368 , piece_id=OPJWV4TDPYNLLY2A5TSKGCUGX542YUX2OMPMILUTPE3TJR4L72AA, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.226822314 (16h3m4.010162413s) [noise=true , address=212.102.52.213:4343 , piece_id=JA6JX5GQOL6WT6MQV4G4UZ2M73QAW5ZII5OW5EO367VKYRESOGGA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.226831161 (16h3m4.01017126s) [noise=true , address=212.102.52.213:4343 , piece_id=JA6JX5GQOL6WT6MQV4G4UZ2M73QAW5ZII5OW5EO367VKYRESOGGA, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.226940960 (16h3m4.010281059s) [noise=true , address=212.102.52.217:4432 , piece_id=GZKMD3YNEMZNXRNHAPFBBXL6T7XOFUHKVJURDCJQ6CTCZRDUW4KQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CPW6Lyzfry6DmU86Fh3rW6u1RS3qpe79mYMaJRB5r4QEEQC6F): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.226946731 (16h3m4.01028683s) [noise=true , address=212.102.52.217:4432 , piece_id=GZKMD3YNEMZNXRNHAPFBBXL6T7XOFUHKVJURDCJQ6CTCZRDUW4KQ, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.227019280 (16h3m4.010359379s) [noise=true , address=212.102.52.216:4416 , piece_id=AMAJVMUOCVS5TOZLHMDOUSQ5YXCFSYEDT7BW4CW56M6VA6LIDJLA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.227026895 (16h3m4.010366994s) [noise=true , address=212.102.52.216:4416 , piece_id=AMAJVMUOCVS5TOZLHMDOUSQ5YXCFSYEDT7BW4CW56M6VA6LIDJLA, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.227043557 (16h3m4.010383656s) [seg_pos={0 0}, upload=15359]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 18:07:18.227113290 (16h3m4.010453389s) [upload=15359]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 18:07:18.227118159 (16h3m4.010458258s) [upload=15359]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 18:07:34.282635746 (16h3m20.065975866s) [upload=15359]: Done uploading segments...
May 25 18:07:34.282653911 (16h3m20.06599402s) [upload=15359]: Upload finished. err: <nil>
Duration: 3.919371433s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
13 out of 60 is failed (threshold=3) [.........X....X.......X......X..XX.XX.X....X.....XX......X..] duration:
7.26285184s trace:
https://jaeger.dev.storj.io/trace/698dfada1f99867a
FAIL (2026-05-25 18:17:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.262866097s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
687.818018ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/7e710747942acdb2
FAIL (2026-05-25 18:16:59 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 687.843816ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [............................X................X..............] duration:
3.859230173s trace:
https://jaeger.dev.storj.io/trace/467f79c1decb330
FAIL (2026-05-25 18:16:50 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
se=true , address=212.102.52.216:4416 , piece_id=AMAJVMUOCVS5TOZLHMDOUSQ5YXCFSYEDT7BW4CW56M6VA6LIDJLA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.813733143 (16h3m2.597073242s) [noise=true , address=212.102.52.213:4340 , piece_id=GGQZRBZNC5DKXOZYV664XOFD2YUAZAL4RPRDQY7ZY6LUN6NKEFKQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.814170728 (16h3m2.597510827s) [noise=true , address=212.102.52.214:4356 , piece_id=5OQLNMQN2NWE2OJO7FS44O6NI4NRMDCGYQQ3OTTH62CQDZZGHVQA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.814532990 (16h3m2.597873089s) [noise=true , address=212.102.52.216:4405 , piece_id=CBCNKWI2VLVCUZDVM54E243ULDQUGYEH5EWRK2ES6V6LRUPWQIIA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.814792664 (16h3m2.598132764s) [noise=true , address=212.102.52.214:4358 , piece_id=NDE6FEYH2OTGRTRIHGJBGJSV77ZUKHXAR4AON3NDKBU33EL4BGZQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.815026731 (16h3m2.59836683s) [noise=true , address=212.102.52.214:4347 , piece_id=6SXGEUYVLSOUSGPUY22UAXQKUWFMWVQOV3NHMMOOJY3NNL5XBAHA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.815255207 (16h3m2.598595296s) [noise=true , address=212.102.52.215:4369 , piece_id=TLNVSD3KWUSXOW3YLEMNRREQW2RMZABFJEEGX64XH7QYAXZBNDHQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.815526655 (16h3m2.598866754s) [noise=true , address=212.102.52.213:4343 , piece_id=JA6JX5GQOL6WT6MQV4G4UZ2M73QAW5ZII5OW5EO367VKYRESOGGA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.817053338 (16h3m2.600393437s) [noise=true , address=212.102.52.213:4328 , piece_id=GUZPGD4HCE5PSNWBWZMLEDHA5AEMZZTEF7N2V4OXC4LCI7TIRIFQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.817184408 (16h3m2.600524507s) [noise=true , address=212.102.52.215:4378 , piece_id=6456XMZDE4RFGHOTOUDM2SNQJDOCHHERTJKKQA6R6KU4EQCD74XA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.818090236 (16h3m2.601430335s) [noise=true , address=212.102.52.216:4408 , piece_id=IEQIPNTTQ3LN7LVVVQSSQU67NLWWG5DC6D5I5W33SKPKPN532DMQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.818269799 (16h3m2.601609898s) [noise=true , address=212.102.52.217:4418 , piece_id=EVBLYS5IQFKBL5TE3UTNM5SENH3COOX6W4JDF53SFFON3JOYQHMQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.819170618 (16h3m2.602510717s) [noise=true , address=212.102.52.217:4432 , piece_id=GZKMD3YNEMZNXRNHAPFBBXL6T7XOFUHKVJURDCJQ6CTCZRDUW4KQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.819559901 (16h3m2.6029s) [noise=true , address=212.102.52.217:4426 , piece_id=CK4UNDBYQQCEVPTT5WGWHBWFH6CRZO3JO7RZL3RO5NOYDFX3DJNA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.820329489 (16h3m2.603669588s) [noise=true , address=212.102.52.217:4431 , piece_id=3JXJF2YLDKTQDXDK666LSTYGYTMHSE3ZXJ7Y5WOUHAXWZVFRBRYA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.821320470 (16h3m2.604660559s) [noise=true , address=212.102.52.215:4388 , piece_id=DNXSHYL6IAMUWAUUVCH6WANAI3HSQIMVJYKSSG3YKLKNS4S6DGJQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:17.875060315 (16h3m3.658400414s) [upload=15359]: Next returned nil segment
May 25 18:07:17.875071437 (16h3m3.658411536s) [upload=15359]: Waiting for error group managing segments...
May 25 18:07:18.161378706 (16h3m3.944718805s) [noise=true , address=212.102.52.214:4356 , piece_id=5OQLNMQN2NWE2OJO7FS44O6NI4NRMDCGYQQ3OTTH62CQDZZGHVQA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.163883035 (16h3m3.947223134s) [noise=true , address=212.102.52.215:4369 , piece_id=TLNVSD3KWUSXOW3YLEMNRREQW2RMZABFJEEGX64XH7QYAXZBNDHQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.169179691 (16h3m3.9525198s) [noise=true , address=212.102.52.214:4347 , piece_id=6SXGEUYVLSOUSGPUY22UAXQKUWFMWVQOV3NHMMOOJY3NNL5XBAHA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.169953397 (16h3m3.953293496s) [noise=true , address=212.102.52.214:4358 , piece_id=NDE6FEYH2OTGRTRIHGJBGJSV77ZUKHXAR4AON3NDKBU33EL4BGZQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.170012670 (16h3m3.953352769s) [noise=true , address=212.102.52.216:4406 , piece_id=DQK2NNDQZWMM7KWBUMYVDJZTGT3SITN3M2ILQ7Y3QNIBV4WZXTTQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.173457273 (16h3m3.956797402s) [noise=true , address=212.102.52.217:4431 , piece_id=3JXJF2YLDKTQDXDK666LSTYGYTMHSE3ZXJ7Y5WOUHAXWZVFRBRYA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.173853839 (16h3m3.957193938s) [noise=true , address=212.102.52.213:4340 , piece_id=GGQZRBZNC5DKXOZYV664XOFD2YUAZAL4RPRDQY7ZY6LUN6NKEFKQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.175372487 (16h3m3.958712586s) [noise=true , address=212.102.52.216:4405 , piece_id=CBCNKWI2VLVCUZDVM54E243ULDQUGYEH5EWRK2ES6V6LRUPWQIIA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.181671326 (16h3m3.965011425s) [noise=true , address=212.102.52.215:4376 , piece_id=RK5KEGAOHFZBU4NN7IO44XPBY5YAOANJ5VLPUU3XVBIGVVFSFZVA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.186238400 (16h3m3.969578509s) [noise=true , address=212.102.52.217:4418 , piece_id=EVBLYS5IQFKBL5TE3UTNM5SENH3COOX6W4JDF53SFFON3JOYQHMQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.190371015 (16h3m3.973711114s) [noise=true , address=212.102.52.213:4328 , piece_id=GUZPGD4HCE5PSNWBWZMLEDHA5AEMZZTEF7N2V4OXC4LCI7TIRIFQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.191589199 (16h3m3.974929298s) [noise=true , address=212.102.52.216:4408 , piece_id=IEQIPNTTQ3LN7LVVVQSSQU67NLWWG5DC6D5I5W33SKPKPN532DMQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.199054834 (16h3m3.982394933s) [noise=true , address=212.102.52.215:4388 , piece_id=DNXSHYL6IAMUWAUUVCH6WANAI3HSQIMVJYKSSG3YKLKNS4S6DGJQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.220275774 (16h3m4.003615873s) [noise=true , address=212.102.52.217:4426 , piece_id=CK4UNDBYQQCEVPTT5WGWHBWFH6CRZO3JO7RZL3RO5NOYDFX3DJNA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.220628726 (16h3m4.003968825s) [noise=true , address=212.102.52.215:4378 , piece_id=6456XMZDE4RFGHOTOUDM2SNQJDOCHHERTJKKQA6R6KU4EQCD74XA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.226546558 (16h3m4.009886657s) [noise=true , address=212.102.52.213:4321 , piece_id=QFBCH5MLCAIFPWMHO4NNQJLPIX7QG5LPQJUHG77WMPI6VPIBQIQA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.226564292 (16h3m4.009904421s) [seg_pos={0 0}, upload=15359]: Satisfied cohort requirements after 16 pieces
May 25 18:07:18.226741169 (16h3m4.010081268s) [noise=true , address=212.102.52.214:4368 , piece_id=OPJWV4TDPYNLLY2A5TSKGCUGX542YUX2OMPMILUTPE3TJR4L72AA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zavf82Jv8HWUWhuGf2nXnhiF1GDupLBVhu6urZQP6N6j71R3C): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.226748514 (16h3m4.010088613s) [noise=true , address=212.102.52.214:4368 , piece_id=OPJWV4TDPYNLLY2A5TSKGCUGX542YUX2OMPMILUTPE3TJR4L72AA, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.226822314 (16h3m4.010162413s) [noise=true , address=212.102.52.213:4343 , piece_id=JA6JX5GQOL6WT6MQV4G4UZ2M73QAW5ZII5OW5EO367VKYRESOGGA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.226831161 (16h3m4.01017126s) [noise=true , address=212.102.52.213:4343 , piece_id=JA6JX5GQOL6WT6MQV4G4UZ2M73QAW5ZII5OW5EO367VKYRESOGGA, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.226940960 (16h3m4.010281059s) [noise=true , address=212.102.52.217:4432 , piece_id=GZKMD3YNEMZNXRNHAPFBBXL6T7XOFUHKVJURDCJQ6CTCZRDUW4KQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CPW6Lyzfry6DmU86Fh3rW6u1RS3qpe79mYMaJRB5r4QEEQC6F): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.226946731 (16h3m4.01028683s) [noise=true , address=212.102.52.217:4432 , piece_id=GZKMD3YNEMZNXRNHAPFBBXL6T7XOFUHKVJURDCJQ6CTCZRDUW4KQ, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.227019280 (16h3m4.010359379s) [noise=true , address=212.102.52.216:4416 , piece_id=AMAJVMUOCVS5TOZLHMDOUSQ5YXCFSYEDT7BW4CW56M6VA6LIDJLA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.227026895 (16h3m4.010366994s) [noise=true , address=212.102.52.216:4416 , piece_id=AMAJVMUOCVS5TOZLHMDOUSQ5YXCFSYEDT7BW4CW56M6VA6LIDJLA, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.227043557 (16h3m4.010383656s) [seg_pos={0 0}, upload=15359]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 18:07:18.227113290 (16h3m4.010453389s) [upload=15359]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 18:07:18.227118159 (16h3m4.010458258s) [upload=15359]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 18:07:34.282635746 (16h3m20.065975866s) [upload=15359]: Done uploading segments...
May 25 18:07:34.282653911 (16h3m20.06599402s) [upload=15359]: Upload finished. err: <nil>
Duration: 3.859281722s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
13 out of 60 is failed (threshold=3) [..........X....X.......X......X..XX.XX.X....X.....XX......X.] duration:
14.972027027s trace:
https://jaeger.dev.storj.io/trace/698dfada1f99867a
FAIL (2026-05-25 18:16:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 14.972037577s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
13 out of 60 is failed (threshold=3) [...........X....X.......X......X..XX.XX.X....X.....XX......X] duration:
1m0.000514719s trace:
https://jaeger.dev.storj.io/trace/698dfada1f99867a
FAIL (2026-05-25 18:16:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000552321s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [.............................X................X.............] duration:
4.028681613s trace:
https://jaeger.dev.storj.io/trace/467f79c1decb330
FAIL (2026-05-25 18:16:24 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
se=true , address=212.102.52.216:4416 , piece_id=AMAJVMUOCVS5TOZLHMDOUSQ5YXCFSYEDT7BW4CW56M6VA6LIDJLA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.813733143 (16h3m2.597073242s) [noise=true , address=212.102.52.213:4340 , piece_id=GGQZRBZNC5DKXOZYV664XOFD2YUAZAL4RPRDQY7ZY6LUN6NKEFKQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.814170728 (16h3m2.597510827s) [noise=true , address=212.102.52.214:4356 , piece_id=5OQLNMQN2NWE2OJO7FS44O6NI4NRMDCGYQQ3OTTH62CQDZZGHVQA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.814532990 (16h3m2.597873089s) [noise=true , address=212.102.52.216:4405 , piece_id=CBCNKWI2VLVCUZDVM54E243ULDQUGYEH5EWRK2ES6V6LRUPWQIIA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.814792664 (16h3m2.598132764s) [noise=true , address=212.102.52.214:4358 , piece_id=NDE6FEYH2OTGRTRIHGJBGJSV77ZUKHXAR4AON3NDKBU33EL4BGZQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.815026731 (16h3m2.59836683s) [noise=true , address=212.102.52.214:4347 , piece_id=6SXGEUYVLSOUSGPUY22UAXQKUWFMWVQOV3NHMMOOJY3NNL5XBAHA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.815255207 (16h3m2.598595296s) [noise=true , address=212.102.52.215:4369 , piece_id=TLNVSD3KWUSXOW3YLEMNRREQW2RMZABFJEEGX64XH7QYAXZBNDHQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.815526655 (16h3m2.598866754s) [noise=true , address=212.102.52.213:4343 , piece_id=JA6JX5GQOL6WT6MQV4G4UZ2M73QAW5ZII5OW5EO367VKYRESOGGA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.817053338 (16h3m2.600393437s) [noise=true , address=212.102.52.213:4328 , piece_id=GUZPGD4HCE5PSNWBWZMLEDHA5AEMZZTEF7N2V4OXC4LCI7TIRIFQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.817184408 (16h3m2.600524507s) [noise=true , address=212.102.52.215:4378 , piece_id=6456XMZDE4RFGHOTOUDM2SNQJDOCHHERTJKKQA6R6KU4EQCD74XA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.818090236 (16h3m2.601430335s) [noise=true , address=212.102.52.216:4408 , piece_id=IEQIPNTTQ3LN7LVVVQSSQU67NLWWG5DC6D5I5W33SKPKPN532DMQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.818269799 (16h3m2.601609898s) [noise=true , address=212.102.52.217:4418 , piece_id=EVBLYS5IQFKBL5TE3UTNM5SENH3COOX6W4JDF53SFFON3JOYQHMQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.819170618 (16h3m2.602510717s) [noise=true , address=212.102.52.217:4432 , piece_id=GZKMD3YNEMZNXRNHAPFBBXL6T7XOFUHKVJURDCJQ6CTCZRDUW4KQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.819559901 (16h3m2.6029s) [noise=true , address=212.102.52.217:4426 , piece_id=CK4UNDBYQQCEVPTT5WGWHBWFH6CRZO3JO7RZL3RO5NOYDFX3DJNA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.820329489 (16h3m2.603669588s) [noise=true , address=212.102.52.217:4431 , piece_id=3JXJF2YLDKTQDXDK666LSTYGYTMHSE3ZXJ7Y5WOUHAXWZVFRBRYA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.821320470 (16h3m2.604660559s) [noise=true , address=212.102.52.215:4388 , piece_id=DNXSHYL6IAMUWAUUVCH6WANAI3HSQIMVJYKSSG3YKLKNS4S6DGJQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:17.875060315 (16h3m3.658400414s) [upload=15359]: Next returned nil segment
May 25 18:07:17.875071437 (16h3m3.658411536s) [upload=15359]: Waiting for error group managing segments...
May 25 18:07:18.161378706 (16h3m3.944718805s) [noise=true , address=212.102.52.214:4356 , piece_id=5OQLNMQN2NWE2OJO7FS44O6NI4NRMDCGYQQ3OTTH62CQDZZGHVQA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.163883035 (16h3m3.947223134s) [noise=true , address=212.102.52.215:4369 , piece_id=TLNVSD3KWUSXOW3YLEMNRREQW2RMZABFJEEGX64XH7QYAXZBNDHQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.169179691 (16h3m3.9525198s) [noise=true , address=212.102.52.214:4347 , piece_id=6SXGEUYVLSOUSGPUY22UAXQKUWFMWVQOV3NHMMOOJY3NNL5XBAHA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.169953397 (16h3m3.953293496s) [noise=true , address=212.102.52.214:4358 , piece_id=NDE6FEYH2OTGRTRIHGJBGJSV77ZUKHXAR4AON3NDKBU33EL4BGZQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.170012670 (16h3m3.953352769s) [noise=true , address=212.102.52.216:4406 , piece_id=DQK2NNDQZWMM7KWBUMYVDJZTGT3SITN3M2ILQ7Y3QNIBV4WZXTTQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.173457273 (16h3m3.956797402s) [noise=true , address=212.102.52.217:4431 , piece_id=3JXJF2YLDKTQDXDK666LSTYGYTMHSE3ZXJ7Y5WOUHAXWZVFRBRYA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.173853839 (16h3m3.957193938s) [noise=true , address=212.102.52.213:4340 , piece_id=GGQZRBZNC5DKXOZYV664XOFD2YUAZAL4RPRDQY7ZY6LUN6NKEFKQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.175372487 (16h3m3.958712586s) [noise=true , address=212.102.52.216:4405 , piece_id=CBCNKWI2VLVCUZDVM54E243ULDQUGYEH5EWRK2ES6V6LRUPWQIIA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.181671326 (16h3m3.965011425s) [noise=true , address=212.102.52.215:4376 , piece_id=RK5KEGAOHFZBU4NN7IO44XPBY5YAOANJ5VLPUU3XVBIGVVFSFZVA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.186238400 (16h3m3.969578509s) [noise=true , address=212.102.52.217:4418 , piece_id=EVBLYS5IQFKBL5TE3UTNM5SENH3COOX6W4JDF53SFFON3JOYQHMQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.190371015 (16h3m3.973711114s) [noise=true , address=212.102.52.213:4328 , piece_id=GUZPGD4HCE5PSNWBWZMLEDHA5AEMZZTEF7N2V4OXC4LCI7TIRIFQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.191589199 (16h3m3.974929298s) [noise=true , address=212.102.52.216:4408 , piece_id=IEQIPNTTQ3LN7LVVVQSSQU67NLWWG5DC6D5I5W33SKPKPN532DMQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.199054834 (16h3m3.982394933s) [noise=true , address=212.102.52.215:4388 , piece_id=DNXSHYL6IAMUWAUUVCH6WANAI3HSQIMVJYKSSG3YKLKNS4S6DGJQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.220275774 (16h3m4.003615873s) [noise=true , address=212.102.52.217:4426 , piece_id=CK4UNDBYQQCEVPTT5WGWHBWFH6CRZO3JO7RZL3RO5NOYDFX3DJNA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.220628726 (16h3m4.003968825s) [noise=true , address=212.102.52.215:4378 , piece_id=6456XMZDE4RFGHOTOUDM2SNQJDOCHHERTJKKQA6R6KU4EQCD74XA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.226546558 (16h3m4.009886657s) [noise=true , address=212.102.52.213:4321 , piece_id=QFBCH5MLCAIFPWMHO4NNQJLPIX7QG5LPQJUHG77WMPI6VPIBQIQA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.226564292 (16h3m4.009904421s) [seg_pos={0 0}, upload=15359]: Satisfied cohort requirements after 16 pieces
May 25 18:07:18.226741169 (16h3m4.010081268s) [noise=true , address=212.102.52.214:4368 , piece_id=OPJWV4TDPYNLLY2A5TSKGCUGX542YUX2OMPMILUTPE3TJR4L72AA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zavf82Jv8HWUWhuGf2nXnhiF1GDupLBVhu6urZQP6N6j71R3C): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.226748514 (16h3m4.010088613s) [noise=true , address=212.102.52.214:4368 , piece_id=OPJWV4TDPYNLLY2A5TSKGCUGX542YUX2OMPMILUTPE3TJR4L72AA, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.226822314 (16h3m4.010162413s) [noise=true , address=212.102.52.213:4343 , piece_id=JA6JX5GQOL6WT6MQV4G4UZ2M73QAW5ZII5OW5EO367VKYRESOGGA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.226831161 (16h3m4.01017126s) [noise=true , address=212.102.52.213:4343 , piece_id=JA6JX5GQOL6WT6MQV4G4UZ2M73QAW5ZII5OW5EO367VKYRESOGGA, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.226940960 (16h3m4.010281059s) [noise=true , address=212.102.52.217:4432 , piece_id=GZKMD3YNEMZNXRNHAPFBBXL6T7XOFUHKVJURDCJQ6CTCZRDUW4KQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CPW6Lyzfry6DmU86Fh3rW6u1RS3qpe79mYMaJRB5r4QEEQC6F): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.226946731 (16h3m4.01028683s) [noise=true , address=212.102.52.217:4432 , piece_id=GZKMD3YNEMZNXRNHAPFBBXL6T7XOFUHKVJURDCJQ6CTCZRDUW4KQ, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.227019280 (16h3m4.010359379s) [noise=true , address=212.102.52.216:4416 , piece_id=AMAJVMUOCVS5TOZLHMDOUSQ5YXCFSYEDT7BW4CW56M6VA6LIDJLA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.227026895 (16h3m4.010366994s) [noise=true , address=212.102.52.216:4416 , piece_id=AMAJVMUOCVS5TOZLHMDOUSQ5YXCFSYEDT7BW4CW56M6VA6LIDJLA, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.227043557 (16h3m4.010383656s) [seg_pos={0 0}, upload=15359]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 18:07:18.227113290 (16h3m4.010453389s) [upload=15359]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 18:07:18.227118159 (16h3m4.010458258s) [upload=15359]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 18:07:34.282635746 (16h3m20.065975866s) [upload=15359]: Done uploading segments...
May 25 18:07:34.282653911 (16h3m20.06599402s) [upload=15359]: Upload finished. err: <nil>
Duration: 4.028704605s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [....................................X.......................] duration:
5.000945577s trace:
https://jaeger.dev.storj.io/trace/14eb8131f680ed75
FAIL (2026-05-25 18:16:15 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000971045s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
673.919097ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/7e710747942acdb2
FAIL (2026-05-25 18:15:49 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 673.940659ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [..............................X................X............] duration:
4.065583674s trace:
https://jaeger.dev.storj.io/trace/467f79c1decb330
FAIL (2026-05-25 18:15:42 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
se=true , address=212.102.52.216:4416 , piece_id=AMAJVMUOCVS5TOZLHMDOUSQ5YXCFSYEDT7BW4CW56M6VA6LIDJLA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.813733143 (16h3m2.597073242s) [noise=true , address=212.102.52.213:4340 , piece_id=GGQZRBZNC5DKXOZYV664XOFD2YUAZAL4RPRDQY7ZY6LUN6NKEFKQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.814170728 (16h3m2.597510827s) [noise=true , address=212.102.52.214:4356 , piece_id=5OQLNMQN2NWE2OJO7FS44O6NI4NRMDCGYQQ3OTTH62CQDZZGHVQA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.814532990 (16h3m2.597873089s) [noise=true , address=212.102.52.216:4405 , piece_id=CBCNKWI2VLVCUZDVM54E243ULDQUGYEH5EWRK2ES6V6LRUPWQIIA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.814792664 (16h3m2.598132764s) [noise=true , address=212.102.52.214:4358 , piece_id=NDE6FEYH2OTGRTRIHGJBGJSV77ZUKHXAR4AON3NDKBU33EL4BGZQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.815026731 (16h3m2.59836683s) [noise=true , address=212.102.52.214:4347 , piece_id=6SXGEUYVLSOUSGPUY22UAXQKUWFMWVQOV3NHMMOOJY3NNL5XBAHA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.815255207 (16h3m2.598595296s) [noise=true , address=212.102.52.215:4369 , piece_id=TLNVSD3KWUSXOW3YLEMNRREQW2RMZABFJEEGX64XH7QYAXZBNDHQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.815526655 (16h3m2.598866754s) [noise=true , address=212.102.52.213:4343 , piece_id=JA6JX5GQOL6WT6MQV4G4UZ2M73QAW5ZII5OW5EO367VKYRESOGGA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.817053338 (16h3m2.600393437s) [noise=true , address=212.102.52.213:4328 , piece_id=GUZPGD4HCE5PSNWBWZMLEDHA5AEMZZTEF7N2V4OXC4LCI7TIRIFQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.817184408 (16h3m2.600524507s) [noise=true , address=212.102.52.215:4378 , piece_id=6456XMZDE4RFGHOTOUDM2SNQJDOCHHERTJKKQA6R6KU4EQCD74XA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.818090236 (16h3m2.601430335s) [noise=true , address=212.102.52.216:4408 , piece_id=IEQIPNTTQ3LN7LVVVQSSQU67NLWWG5DC6D5I5W33SKPKPN532DMQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.818269799 (16h3m2.601609898s) [noise=true , address=212.102.52.217:4418 , piece_id=EVBLYS5IQFKBL5TE3UTNM5SENH3COOX6W4JDF53SFFON3JOYQHMQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.819170618 (16h3m2.602510717s) [noise=true , address=212.102.52.217:4432 , piece_id=GZKMD3YNEMZNXRNHAPFBBXL6T7XOFUHKVJURDCJQ6CTCZRDUW4KQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.819559901 (16h3m2.6029s) [noise=true , address=212.102.52.217:4426 , piece_id=CK4UNDBYQQCEVPTT5WGWHBWFH6CRZO3JO7RZL3RO5NOYDFX3DJNA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.820329489 (16h3m2.603669588s) [noise=true , address=212.102.52.217:4431 , piece_id=3JXJF2YLDKTQDXDK666LSTYGYTMHSE3ZXJ7Y5WOUHAXWZVFRBRYA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.821320470 (16h3m2.604660559s) [noise=true , address=212.102.52.215:4388 , piece_id=DNXSHYL6IAMUWAUUVCH6WANAI3HSQIMVJYKSSG3YKLKNS4S6DGJQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:17.875060315 (16h3m3.658400414s) [upload=15359]: Next returned nil segment
May 25 18:07:17.875071437 (16h3m3.658411536s) [upload=15359]: Waiting for error group managing segments...
May 25 18:07:18.161378706 (16h3m3.944718805s) [noise=true , address=212.102.52.214:4356 , piece_id=5OQLNMQN2NWE2OJO7FS44O6NI4NRMDCGYQQ3OTTH62CQDZZGHVQA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.163883035 (16h3m3.947223134s) [noise=true , address=212.102.52.215:4369 , piece_id=TLNVSD3KWUSXOW3YLEMNRREQW2RMZABFJEEGX64XH7QYAXZBNDHQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.169179691 (16h3m3.9525198s) [noise=true , address=212.102.52.214:4347 , piece_id=6SXGEUYVLSOUSGPUY22UAXQKUWFMWVQOV3NHMMOOJY3NNL5XBAHA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.169953397 (16h3m3.953293496s) [noise=true , address=212.102.52.214:4358 , piece_id=NDE6FEYH2OTGRTRIHGJBGJSV77ZUKHXAR4AON3NDKBU33EL4BGZQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.170012670 (16h3m3.953352769s) [noise=true , address=212.102.52.216:4406 , piece_id=DQK2NNDQZWMM7KWBUMYVDJZTGT3SITN3M2ILQ7Y3QNIBV4WZXTTQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.173457273 (16h3m3.956797402s) [noise=true , address=212.102.52.217:4431 , piece_id=3JXJF2YLDKTQDXDK666LSTYGYTMHSE3ZXJ7Y5WOUHAXWZVFRBRYA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.173853839 (16h3m3.957193938s) [noise=true , address=212.102.52.213:4340 , piece_id=GGQZRBZNC5DKXOZYV664XOFD2YUAZAL4RPRDQY7ZY6LUN6NKEFKQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.175372487 (16h3m3.958712586s) [noise=true , address=212.102.52.216:4405 , piece_id=CBCNKWI2VLVCUZDVM54E243ULDQUGYEH5EWRK2ES6V6LRUPWQIIA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.181671326 (16h3m3.965011425s) [noise=true , address=212.102.52.215:4376 , piece_id=RK5KEGAOHFZBU4NN7IO44XPBY5YAOANJ5VLPUU3XVBIGVVFSFZVA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.186238400 (16h3m3.969578509s) [noise=true , address=212.102.52.217:4418 , piece_id=EVBLYS5IQFKBL5TE3UTNM5SENH3COOX6W4JDF53SFFON3JOYQHMQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.190371015 (16h3m3.973711114s) [noise=true , address=212.102.52.213:4328 , piece_id=GUZPGD4HCE5PSNWBWZMLEDHA5AEMZZTEF7N2V4OXC4LCI7TIRIFQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.191589199 (16h3m3.974929298s) [noise=true , address=212.102.52.216:4408 , piece_id=IEQIPNTTQ3LN7LVVVQSSQU67NLWWG5DC6D5I5W33SKPKPN532DMQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.199054834 (16h3m3.982394933s) [noise=true , address=212.102.52.215:4388 , piece_id=DNXSHYL6IAMUWAUUVCH6WANAI3HSQIMVJYKSSG3YKLKNS4S6DGJQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.220275774 (16h3m4.003615873s) [noise=true , address=212.102.52.217:4426 , piece_id=CK4UNDBYQQCEVPTT5WGWHBWFH6CRZO3JO7RZL3RO5NOYDFX3DJNA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.220628726 (16h3m4.003968825s) [noise=true , address=212.102.52.215:4378 , piece_id=6456XMZDE4RFGHOTOUDM2SNQJDOCHHERTJKKQA6R6KU4EQCD74XA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.226546558 (16h3m4.009886657s) [noise=true , address=212.102.52.213:4321 , piece_id=QFBCH5MLCAIFPWMHO4NNQJLPIX7QG5LPQJUHG77WMPI6VPIBQIQA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.226564292 (16h3m4.009904421s) [seg_pos={0 0}, upload=15359]: Satisfied cohort requirements after 16 pieces
May 25 18:07:18.226741169 (16h3m4.010081268s) [noise=true , address=212.102.52.214:4368 , piece_id=OPJWV4TDPYNLLY2A5TSKGCUGX542YUX2OMPMILUTPE3TJR4L72AA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zavf82Jv8HWUWhuGf2nXnhiF1GDupLBVhu6urZQP6N6j71R3C): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.226748514 (16h3m4.010088613s) [noise=true , address=212.102.52.214:4368 , piece_id=OPJWV4TDPYNLLY2A5TSKGCUGX542YUX2OMPMILUTPE3TJR4L72AA, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.226822314 (16h3m4.010162413s) [noise=true , address=212.102.52.213:4343 , piece_id=JA6JX5GQOL6WT6MQV4G4UZ2M73QAW5ZII5OW5EO367VKYRESOGGA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.226831161 (16h3m4.01017126s) [noise=true , address=212.102.52.213:4343 , piece_id=JA6JX5GQOL6WT6MQV4G4UZ2M73QAW5ZII5OW5EO367VKYRESOGGA, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.226940960 (16h3m4.010281059s) [noise=true , address=212.102.52.217:4432 , piece_id=GZKMD3YNEMZNXRNHAPFBBXL6T7XOFUHKVJURDCJQ6CTCZRDUW4KQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CPW6Lyzfry6DmU86Fh3rW6u1RS3qpe79mYMaJRB5r4QEEQC6F): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.226946731 (16h3m4.01028683s) [noise=true , address=212.102.52.217:4432 , piece_id=GZKMD3YNEMZNXRNHAPFBBXL6T7XOFUHKVJURDCJQ6CTCZRDUW4KQ, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.227019280 (16h3m4.010359379s) [noise=true , address=212.102.52.216:4416 , piece_id=AMAJVMUOCVS5TOZLHMDOUSQ5YXCFSYEDT7BW4CW56M6VA6LIDJLA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.227026895 (16h3m4.010366994s) [noise=true , address=212.102.52.216:4416 , piece_id=AMAJVMUOCVS5TOZLHMDOUSQ5YXCFSYEDT7BW4CW56M6VA6LIDJLA, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.227043557 (16h3m4.010383656s) [seg_pos={0 0}, upload=15359]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 18:07:18.227113290 (16h3m4.010453389s) [upload=15359]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 18:07:18.227118159 (16h3m4.010458258s) [upload=15359]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 18:07:34.282635746 (16h3m20.065975866s) [upload=15359]: Done uploading segments...
May 25 18:07:34.282653911 (16h3m20.06599402s) [upload=15359]: Upload finished. err: <nil>
Duration: 4.065608259s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.....................................X......................] duration:
3.89516256s trace:
https://jaeger.dev.storj.io/trace/14eb8131f680ed75
FAIL (2026-05-25 18:15:32 UTC)
orbiter warning: context deadline exceeded
Duration: 3.895188681s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
675.743639ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/7e710747942acdb2
FAIL (2026-05-25 18:15:22 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 675.762976ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [...............................X................X...........] duration:
3.958022577s trace:
https://jaeger.dev.storj.io/trace/467f79c1decb330
FAIL (2026-05-25 18:15:08 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
se=true , address=212.102.52.216:4416 , piece_id=AMAJVMUOCVS5TOZLHMDOUSQ5YXCFSYEDT7BW4CW56M6VA6LIDJLA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.813733143 (16h3m2.597073242s) [noise=true , address=212.102.52.213:4340 , piece_id=GGQZRBZNC5DKXOZYV664XOFD2YUAZAL4RPRDQY7ZY6LUN6NKEFKQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.814170728 (16h3m2.597510827s) [noise=true , address=212.102.52.214:4356 , piece_id=5OQLNMQN2NWE2OJO7FS44O6NI4NRMDCGYQQ3OTTH62CQDZZGHVQA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.814532990 (16h3m2.597873089s) [noise=true , address=212.102.52.216:4405 , piece_id=CBCNKWI2VLVCUZDVM54E243ULDQUGYEH5EWRK2ES6V6LRUPWQIIA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.814792664 (16h3m2.598132764s) [noise=true , address=212.102.52.214:4358 , piece_id=NDE6FEYH2OTGRTRIHGJBGJSV77ZUKHXAR4AON3NDKBU33EL4BGZQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.815026731 (16h3m2.59836683s) [noise=true , address=212.102.52.214:4347 , piece_id=6SXGEUYVLSOUSGPUY22UAXQKUWFMWVQOV3NHMMOOJY3NNL5XBAHA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.815255207 (16h3m2.598595296s) [noise=true , address=212.102.52.215:4369 , piece_id=TLNVSD3KWUSXOW3YLEMNRREQW2RMZABFJEEGX64XH7QYAXZBNDHQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.815526655 (16h3m2.598866754s) [noise=true , address=212.102.52.213:4343 , piece_id=JA6JX5GQOL6WT6MQV4G4UZ2M73QAW5ZII5OW5EO367VKYRESOGGA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.817053338 (16h3m2.600393437s) [noise=true , address=212.102.52.213:4328 , piece_id=GUZPGD4HCE5PSNWBWZMLEDHA5AEMZZTEF7N2V4OXC4LCI7TIRIFQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.817184408 (16h3m2.600524507s) [noise=true , address=212.102.52.215:4378 , piece_id=6456XMZDE4RFGHOTOUDM2SNQJDOCHHERTJKKQA6R6KU4EQCD74XA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.818090236 (16h3m2.601430335s) [noise=true , address=212.102.52.216:4408 , piece_id=IEQIPNTTQ3LN7LVVVQSSQU67NLWWG5DC6D5I5W33SKPKPN532DMQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.818269799 (16h3m2.601609898s) [noise=true , address=212.102.52.217:4418 , piece_id=EVBLYS5IQFKBL5TE3UTNM5SENH3COOX6W4JDF53SFFON3JOYQHMQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.819170618 (16h3m2.602510717s) [noise=true , address=212.102.52.217:4432 , piece_id=GZKMD3YNEMZNXRNHAPFBBXL6T7XOFUHKVJURDCJQ6CTCZRDUW4KQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.819559901 (16h3m2.6029s) [noise=true , address=212.102.52.217:4426 , piece_id=CK4UNDBYQQCEVPTT5WGWHBWFH6CRZO3JO7RZL3RO5NOYDFX3DJNA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.820329489 (16h3m2.603669588s) [noise=true , address=212.102.52.217:4431 , piece_id=3JXJF2YLDKTQDXDK666LSTYGYTMHSE3ZXJ7Y5WOUHAXWZVFRBRYA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.821320470 (16h3m2.604660559s) [noise=true , address=212.102.52.215:4388 , piece_id=DNXSHYL6IAMUWAUUVCH6WANAI3HSQIMVJYKSSG3YKLKNS4S6DGJQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:17.875060315 (16h3m3.658400414s) [upload=15359]: Next returned nil segment
May 25 18:07:17.875071437 (16h3m3.658411536s) [upload=15359]: Waiting for error group managing segments...
May 25 18:07:18.161378706 (16h3m3.944718805s) [noise=true , address=212.102.52.214:4356 , piece_id=5OQLNMQN2NWE2OJO7FS44O6NI4NRMDCGYQQ3OTTH62CQDZZGHVQA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.163883035 (16h3m3.947223134s) [noise=true , address=212.102.52.215:4369 , piece_id=TLNVSD3KWUSXOW3YLEMNRREQW2RMZABFJEEGX64XH7QYAXZBNDHQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.169179691 (16h3m3.9525198s) [noise=true , address=212.102.52.214:4347 , piece_id=6SXGEUYVLSOUSGPUY22UAXQKUWFMWVQOV3NHMMOOJY3NNL5XBAHA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.169953397 (16h3m3.953293496s) [noise=true , address=212.102.52.214:4358 , piece_id=NDE6FEYH2OTGRTRIHGJBGJSV77ZUKHXAR4AON3NDKBU33EL4BGZQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.170012670 (16h3m3.953352769s) [noise=true , address=212.102.52.216:4406 , piece_id=DQK2NNDQZWMM7KWBUMYVDJZTGT3SITN3M2ILQ7Y3QNIBV4WZXTTQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.173457273 (16h3m3.956797402s) [noise=true , address=212.102.52.217:4431 , piece_id=3JXJF2YLDKTQDXDK666LSTYGYTMHSE3ZXJ7Y5WOUHAXWZVFRBRYA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.173853839 (16h3m3.957193938s) [noise=true , address=212.102.52.213:4340 , piece_id=GGQZRBZNC5DKXOZYV664XOFD2YUAZAL4RPRDQY7ZY6LUN6NKEFKQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.175372487 (16h3m3.958712586s) [noise=true , address=212.102.52.216:4405 , piece_id=CBCNKWI2VLVCUZDVM54E243ULDQUGYEH5EWRK2ES6V6LRUPWQIIA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.181671326 (16h3m3.965011425s) [noise=true , address=212.102.52.215:4376 , piece_id=RK5KEGAOHFZBU4NN7IO44XPBY5YAOANJ5VLPUU3XVBIGVVFSFZVA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.186238400 (16h3m3.969578509s) [noise=true , address=212.102.52.217:4418 , piece_id=EVBLYS5IQFKBL5TE3UTNM5SENH3COOX6W4JDF53SFFON3JOYQHMQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.190371015 (16h3m3.973711114s) [noise=true , address=212.102.52.213:4328 , piece_id=GUZPGD4HCE5PSNWBWZMLEDHA5AEMZZTEF7N2V4OXC4LCI7TIRIFQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.191589199 (16h3m3.974929298s) [noise=true , address=212.102.52.216:4408 , piece_id=IEQIPNTTQ3LN7LVVVQSSQU67NLWWG5DC6D5I5W33SKPKPN532DMQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.199054834 (16h3m3.982394933s) [noise=true , address=212.102.52.215:4388 , piece_id=DNXSHYL6IAMUWAUUVCH6WANAI3HSQIMVJYKSSG3YKLKNS4S6DGJQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.220275774 (16h3m4.003615873s) [noise=true , address=212.102.52.217:4426 , piece_id=CK4UNDBYQQCEVPTT5WGWHBWFH6CRZO3JO7RZL3RO5NOYDFX3DJNA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.220628726 (16h3m4.003968825s) [noise=true , address=212.102.52.215:4378 , piece_id=6456XMZDE4RFGHOTOUDM2SNQJDOCHHERTJKKQA6R6KU4EQCD74XA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.226546558 (16h3m4.009886657s) [noise=true , address=212.102.52.213:4321 , piece_id=QFBCH5MLCAIFPWMHO4NNQJLPIX7QG5LPQJUHG77WMPI6VPIBQIQA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.226564292 (16h3m4.009904421s) [seg_pos={0 0}, upload=15359]: Satisfied cohort requirements after 16 pieces
May 25 18:07:18.226741169 (16h3m4.010081268s) [noise=true , address=212.102.52.214:4368 , piece_id=OPJWV4TDPYNLLY2A5TSKGCUGX542YUX2OMPMILUTPE3TJR4L72AA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zavf82Jv8HWUWhuGf2nXnhiF1GDupLBVhu6urZQP6N6j71R3C): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.226748514 (16h3m4.010088613s) [noise=true , address=212.102.52.214:4368 , piece_id=OPJWV4TDPYNLLY2A5TSKGCUGX542YUX2OMPMILUTPE3TJR4L72AA, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.226822314 (16h3m4.010162413s) [noise=true , address=212.102.52.213:4343 , piece_id=JA6JX5GQOL6WT6MQV4G4UZ2M73QAW5ZII5OW5EO367VKYRESOGGA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.226831161 (16h3m4.01017126s) [noise=true , address=212.102.52.213:4343 , piece_id=JA6JX5GQOL6WT6MQV4G4UZ2M73QAW5ZII5OW5EO367VKYRESOGGA, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.226940960 (16h3m4.010281059s) [noise=true , address=212.102.52.217:4432 , piece_id=GZKMD3YNEMZNXRNHAPFBBXL6T7XOFUHKVJURDCJQ6CTCZRDUW4KQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CPW6Lyzfry6DmU86Fh3rW6u1RS3qpe79mYMaJRB5r4QEEQC6F): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.226946731 (16h3m4.01028683s) [noise=true , address=212.102.52.217:4432 , piece_id=GZKMD3YNEMZNXRNHAPFBBXL6T7XOFUHKVJURDCJQ6CTCZRDUW4KQ, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.227019280 (16h3m4.010359379s) [noise=true , address=212.102.52.216:4416 , piece_id=AMAJVMUOCVS5TOZLHMDOUSQ5YXCFSYEDT7BW4CW56M6VA6LIDJLA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.227026895 (16h3m4.010366994s) [noise=true , address=212.102.52.216:4416 , piece_id=AMAJVMUOCVS5TOZLHMDOUSQ5YXCFSYEDT7BW4CW56M6VA6LIDJLA, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.227043557 (16h3m4.010383656s) [seg_pos={0 0}, upload=15359]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 18:07:18.227113290 (16h3m4.010453389s) [upload=15359]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 18:07:18.227118159 (16h3m4.010458258s) [upload=15359]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 18:07:34.282635746 (16h3m20.065975866s) [upload=15359]: Done uploading segments...
May 25 18:07:34.282653911 (16h3m20.06599402s) [upload=15359]: Upload finished. err: <nil>
Duration: 3.958048556s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
678.413ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/7e710747942acdb2
FAIL (2026-05-25 18:15:02 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 678.438749ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [................................X................X..........] duration:
22.155650823s trace:
https://jaeger.dev.storj.io/trace/467f79c1decb330
FAIL (2026-05-25 18:14:59 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
se=true , address=212.102.52.216:4416 , piece_id=AMAJVMUOCVS5TOZLHMDOUSQ5YXCFSYEDT7BW4CW56M6VA6LIDJLA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.813733143 (16h3m2.597073242s) [noise=true , address=212.102.52.213:4340 , piece_id=GGQZRBZNC5DKXOZYV664XOFD2YUAZAL4RPRDQY7ZY6LUN6NKEFKQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.814170728 (16h3m2.597510827s) [noise=true , address=212.102.52.214:4356 , piece_id=5OQLNMQN2NWE2OJO7FS44O6NI4NRMDCGYQQ3OTTH62CQDZZGHVQA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.814532990 (16h3m2.597873089s) [noise=true , address=212.102.52.216:4405 , piece_id=CBCNKWI2VLVCUZDVM54E243ULDQUGYEH5EWRK2ES6V6LRUPWQIIA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.814792664 (16h3m2.598132764s) [noise=true , address=212.102.52.214:4358 , piece_id=NDE6FEYH2OTGRTRIHGJBGJSV77ZUKHXAR4AON3NDKBU33EL4BGZQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.815026731 (16h3m2.59836683s) [noise=true , address=212.102.52.214:4347 , piece_id=6SXGEUYVLSOUSGPUY22UAXQKUWFMWVQOV3NHMMOOJY3NNL5XBAHA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.815255207 (16h3m2.598595296s) [noise=true , address=212.102.52.215:4369 , piece_id=TLNVSD3KWUSXOW3YLEMNRREQW2RMZABFJEEGX64XH7QYAXZBNDHQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.815526655 (16h3m2.598866754s) [noise=true , address=212.102.52.213:4343 , piece_id=JA6JX5GQOL6WT6MQV4G4UZ2M73QAW5ZII5OW5EO367VKYRESOGGA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.817053338 (16h3m2.600393437s) [noise=true , address=212.102.52.213:4328 , piece_id=GUZPGD4HCE5PSNWBWZMLEDHA5AEMZZTEF7N2V4OXC4LCI7TIRIFQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.817184408 (16h3m2.600524507s) [noise=true , address=212.102.52.215:4378 , piece_id=6456XMZDE4RFGHOTOUDM2SNQJDOCHHERTJKKQA6R6KU4EQCD74XA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.818090236 (16h3m2.601430335s) [noise=true , address=212.102.52.216:4408 , piece_id=IEQIPNTTQ3LN7LVVVQSSQU67NLWWG5DC6D5I5W33SKPKPN532DMQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.818269799 (16h3m2.601609898s) [noise=true , address=212.102.52.217:4418 , piece_id=EVBLYS5IQFKBL5TE3UTNM5SENH3COOX6W4JDF53SFFON3JOYQHMQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.819170618 (16h3m2.602510717s) [noise=true , address=212.102.52.217:4432 , piece_id=GZKMD3YNEMZNXRNHAPFBBXL6T7XOFUHKVJURDCJQ6CTCZRDUW4KQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.819559901 (16h3m2.6029s) [noise=true , address=212.102.52.217:4426 , piece_id=CK4UNDBYQQCEVPTT5WGWHBWFH6CRZO3JO7RZL3RO5NOYDFX3DJNA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.820329489 (16h3m2.603669588s) [noise=true , address=212.102.52.217:4431 , piece_id=3JXJF2YLDKTQDXDK666LSTYGYTMHSE3ZXJ7Y5WOUHAXWZVFRBRYA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.821320470 (16h3m2.604660559s) [noise=true , address=212.102.52.215:4388 , piece_id=DNXSHYL6IAMUWAUUVCH6WANAI3HSQIMVJYKSSG3YKLKNS4S6DGJQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:17.875060315 (16h3m3.658400414s) [upload=15359]: Next returned nil segment
May 25 18:07:17.875071437 (16h3m3.658411536s) [upload=15359]: Waiting for error group managing segments...
May 25 18:07:18.161378706 (16h3m3.944718805s) [noise=true , address=212.102.52.214:4356 , piece_id=5OQLNMQN2NWE2OJO7FS44O6NI4NRMDCGYQQ3OTTH62CQDZZGHVQA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.163883035 (16h3m3.947223134s) [noise=true , address=212.102.52.215:4369 , piece_id=TLNVSD3KWUSXOW3YLEMNRREQW2RMZABFJEEGX64XH7QYAXZBNDHQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.169179691 (16h3m3.9525198s) [noise=true , address=212.102.52.214:4347 , piece_id=6SXGEUYVLSOUSGPUY22UAXQKUWFMWVQOV3NHMMOOJY3NNL5XBAHA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.169953397 (16h3m3.953293496s) [noise=true , address=212.102.52.214:4358 , piece_id=NDE6FEYH2OTGRTRIHGJBGJSV77ZUKHXAR4AON3NDKBU33EL4BGZQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.170012670 (16h3m3.953352769s) [noise=true , address=212.102.52.216:4406 , piece_id=DQK2NNDQZWMM7KWBUMYVDJZTGT3SITN3M2ILQ7Y3QNIBV4WZXTTQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.173457273 (16h3m3.956797402s) [noise=true , address=212.102.52.217:4431 , piece_id=3JXJF2YLDKTQDXDK666LSTYGYTMHSE3ZXJ7Y5WOUHAXWZVFRBRYA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.173853839 (16h3m3.957193938s) [noise=true , address=212.102.52.213:4340 , piece_id=GGQZRBZNC5DKXOZYV664XOFD2YUAZAL4RPRDQY7ZY6LUN6NKEFKQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.175372487 (16h3m3.958712586s) [noise=true , address=212.102.52.216:4405 , piece_id=CBCNKWI2VLVCUZDVM54E243ULDQUGYEH5EWRK2ES6V6LRUPWQIIA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.181671326 (16h3m3.965011425s) [noise=true , address=212.102.52.215:4376 , piece_id=RK5KEGAOHFZBU4NN7IO44XPBY5YAOANJ5VLPUU3XVBIGVVFSFZVA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.186238400 (16h3m3.969578509s) [noise=true , address=212.102.52.217:4418 , piece_id=EVBLYS5IQFKBL5TE3UTNM5SENH3COOX6W4JDF53SFFON3JOYQHMQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.190371015 (16h3m3.973711114s) [noise=true , address=212.102.52.213:4328 , piece_id=GUZPGD4HCE5PSNWBWZMLEDHA5AEMZZTEF7N2V4OXC4LCI7TIRIFQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.191589199 (16h3m3.974929298s) [noise=true , address=212.102.52.216:4408 , piece_id=IEQIPNTTQ3LN7LVVVQSSQU67NLWWG5DC6D5I5W33SKPKPN532DMQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.199054834 (16h3m3.982394933s) [noise=true , address=212.102.52.215:4388 , piece_id=DNXSHYL6IAMUWAUUVCH6WANAI3HSQIMVJYKSSG3YKLKNS4S6DGJQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.220275774 (16h3m4.003615873s) [noise=true , address=212.102.52.217:4426 , piece_id=CK4UNDBYQQCEVPTT5WGWHBWFH6CRZO3JO7RZL3RO5NOYDFX3DJNA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.220628726 (16h3m4.003968825s) [noise=true , address=212.102.52.215:4378 , piece_id=6456XMZDE4RFGHOTOUDM2SNQJDOCHHERTJKKQA6R6KU4EQCD74XA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.226546558 (16h3m4.009886657s) [noise=true , address=212.102.52.213:4321 , piece_id=QFBCH5MLCAIFPWMHO4NNQJLPIX7QG5LPQJUHG77WMPI6VPIBQIQA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.226564292 (16h3m4.009904421s) [seg_pos={0 0}, upload=15359]: Satisfied cohort requirements after 16 pieces
May 25 18:07:18.226741169 (16h3m4.010081268s) [noise=true , address=212.102.52.214:4368 , piece_id=OPJWV4TDPYNLLY2A5TSKGCUGX542YUX2OMPMILUTPE3TJR4L72AA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zavf82Jv8HWUWhuGf2nXnhiF1GDupLBVhu6urZQP6N6j71R3C): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.226748514 (16h3m4.010088613s) [noise=true , address=212.102.52.214:4368 , piece_id=OPJWV4TDPYNLLY2A5TSKGCUGX542YUX2OMPMILUTPE3TJR4L72AA, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.226822314 (16h3m4.010162413s) [noise=true , address=212.102.52.213:4343 , piece_id=JA6JX5GQOL6WT6MQV4G4UZ2M73QAW5ZII5OW5EO367VKYRESOGGA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.226831161 (16h3m4.01017126s) [noise=true , address=212.102.52.213:4343 , piece_id=JA6JX5GQOL6WT6MQV4G4UZ2M73QAW5ZII5OW5EO367VKYRESOGGA, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.226940960 (16h3m4.010281059s) [noise=true , address=212.102.52.217:4432 , piece_id=GZKMD3YNEMZNXRNHAPFBBXL6T7XOFUHKVJURDCJQ6CTCZRDUW4KQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CPW6Lyzfry6DmU86Fh3rW6u1RS3qpe79mYMaJRB5r4QEEQC6F): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.226946731 (16h3m4.01028683s) [noise=true , address=212.102.52.217:4432 , piece_id=GZKMD3YNEMZNXRNHAPFBBXL6T7XOFUHKVJURDCJQ6CTCZRDUW4KQ, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.227019280 (16h3m4.010359379s) [noise=true , address=212.102.52.216:4416 , piece_id=AMAJVMUOCVS5TOZLHMDOUSQ5YXCFSYEDT7BW4CW56M6VA6LIDJLA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.227026895 (16h3m4.010366994s) [noise=true , address=212.102.52.216:4416 , piece_id=AMAJVMUOCVS5TOZLHMDOUSQ5YXCFSYEDT7BW4CW56M6VA6LIDJLA, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.227043557 (16h3m4.010383656s) [seg_pos={0 0}, upload=15359]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 18:07:18.227113290 (16h3m4.010453389s) [upload=15359]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 18:07:18.227118159 (16h3m4.010458258s) [upload=15359]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 18:07:34.282635746 (16h3m20.065975866s) [upload=15359]: Done uploading segments...
May 25 18:07:34.282653911 (16h3m20.06599402s) [upload=15359]: Upload finished. err: <nil>
Duration: 22.155672013s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
681.617417ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/7e710747942acdb2
FAIL (2026-05-25 18:14:46 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 681.628828ms
core
satellite
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
12 out of 60 is failed (threshold=3) [............X....X.......X......X..XX.XX.X....X.....XX......] duration:
6.908224168s trace:
https://jaeger.dev.storj.io/trace/59b9e788be2903db
FAIL (2026-05-25 18:14:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.908247604s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
678.730667ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/7e710747942acdb2
FAIL (2026-05-25 18:14:27 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 678.755616ms
core
satellite
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [......................................X.....................] duration:
3.748378893s trace:
https://jaeger.dev.storj.io/trace/14eb8131f680ed75
FAIL (2026-05-25 18:14:19 UTC)
orbiter warning: context deadline exceeded
Duration: 3.74840358s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
12 out of 60 is failed (threshold=3) [.............X....X.......X......X..XX.XX.X....X.....XX.....] duration:
6.638687862s trace:
https://jaeger.dev.storj.io/trace/59b9e788be2903db
FAIL (2026-05-25 18:14:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.638711526s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
674.918926ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/7e710747942acdb2
FAIL (2026-05-25 18:14:07 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 674.938724ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [.................................X................X.........] duration:
4.301074329s trace:
https://jaeger.dev.storj.io/trace/467f79c1decb330
FAIL (2026-05-25 18:13:57 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
se=true , address=212.102.52.216:4416 , piece_id=AMAJVMUOCVS5TOZLHMDOUSQ5YXCFSYEDT7BW4CW56M6VA6LIDJLA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.813733143 (16h3m2.597073242s) [noise=true , address=212.102.52.213:4340 , piece_id=GGQZRBZNC5DKXOZYV664XOFD2YUAZAL4RPRDQY7ZY6LUN6NKEFKQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.814170728 (16h3m2.597510827s) [noise=true , address=212.102.52.214:4356 , piece_id=5OQLNMQN2NWE2OJO7FS44O6NI4NRMDCGYQQ3OTTH62CQDZZGHVQA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.814532990 (16h3m2.597873089s) [noise=true , address=212.102.52.216:4405 , piece_id=CBCNKWI2VLVCUZDVM54E243ULDQUGYEH5EWRK2ES6V6LRUPWQIIA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.814792664 (16h3m2.598132764s) [noise=true , address=212.102.52.214:4358 , piece_id=NDE6FEYH2OTGRTRIHGJBGJSV77ZUKHXAR4AON3NDKBU33EL4BGZQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.815026731 (16h3m2.59836683s) [noise=true , address=212.102.52.214:4347 , piece_id=6SXGEUYVLSOUSGPUY22UAXQKUWFMWVQOV3NHMMOOJY3NNL5XBAHA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.815255207 (16h3m2.598595296s) [noise=true , address=212.102.52.215:4369 , piece_id=TLNVSD3KWUSXOW3YLEMNRREQW2RMZABFJEEGX64XH7QYAXZBNDHQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.815526655 (16h3m2.598866754s) [noise=true , address=212.102.52.213:4343 , piece_id=JA6JX5GQOL6WT6MQV4G4UZ2M73QAW5ZII5OW5EO367VKYRESOGGA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.817053338 (16h3m2.600393437s) [noise=true , address=212.102.52.213:4328 , piece_id=GUZPGD4HCE5PSNWBWZMLEDHA5AEMZZTEF7N2V4OXC4LCI7TIRIFQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.817184408 (16h3m2.600524507s) [noise=true , address=212.102.52.215:4378 , piece_id=6456XMZDE4RFGHOTOUDM2SNQJDOCHHERTJKKQA6R6KU4EQCD74XA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.818090236 (16h3m2.601430335s) [noise=true , address=212.102.52.216:4408 , piece_id=IEQIPNTTQ3LN7LVVVQSSQU67NLWWG5DC6D5I5W33SKPKPN532DMQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.818269799 (16h3m2.601609898s) [noise=true , address=212.102.52.217:4418 , piece_id=EVBLYS5IQFKBL5TE3UTNM5SENH3COOX6W4JDF53SFFON3JOYQHMQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.819170618 (16h3m2.602510717s) [noise=true , address=212.102.52.217:4432 , piece_id=GZKMD3YNEMZNXRNHAPFBBXL6T7XOFUHKVJURDCJQ6CTCZRDUW4KQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.819559901 (16h3m2.6029s) [noise=true , address=212.102.52.217:4426 , piece_id=CK4UNDBYQQCEVPTT5WGWHBWFH6CRZO3JO7RZL3RO5NOYDFX3DJNA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.820329489 (16h3m2.603669588s) [noise=true , address=212.102.52.217:4431 , piece_id=3JXJF2YLDKTQDXDK666LSTYGYTMHSE3ZXJ7Y5WOUHAXWZVFRBRYA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.821320470 (16h3m2.604660559s) [noise=true , address=212.102.52.215:4388 , piece_id=DNXSHYL6IAMUWAUUVCH6WANAI3HSQIMVJYKSSG3YKLKNS4S6DGJQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:17.875060315 (16h3m3.658400414s) [upload=15359]: Next returned nil segment
May 25 18:07:17.875071437 (16h3m3.658411536s) [upload=15359]: Waiting for error group managing segments...
May 25 18:07:18.161378706 (16h3m3.944718805s) [noise=true , address=212.102.52.214:4356 , piece_id=5OQLNMQN2NWE2OJO7FS44O6NI4NRMDCGYQQ3OTTH62CQDZZGHVQA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.163883035 (16h3m3.947223134s) [noise=true , address=212.102.52.215:4369 , piece_id=TLNVSD3KWUSXOW3YLEMNRREQW2RMZABFJEEGX64XH7QYAXZBNDHQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.169179691 (16h3m3.9525198s) [noise=true , address=212.102.52.214:4347 , piece_id=6SXGEUYVLSOUSGPUY22UAXQKUWFMWVQOV3NHMMOOJY3NNL5XBAHA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.169953397 (16h3m3.953293496s) [noise=true , address=212.102.52.214:4358 , piece_id=NDE6FEYH2OTGRTRIHGJBGJSV77ZUKHXAR4AON3NDKBU33EL4BGZQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.170012670 (16h3m3.953352769s) [noise=true , address=212.102.52.216:4406 , piece_id=DQK2NNDQZWMM7KWBUMYVDJZTGT3SITN3M2ILQ7Y3QNIBV4WZXTTQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.173457273 (16h3m3.956797402s) [noise=true , address=212.102.52.217:4431 , piece_id=3JXJF2YLDKTQDXDK666LSTYGYTMHSE3ZXJ7Y5WOUHAXWZVFRBRYA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.173853839 (16h3m3.957193938s) [noise=true , address=212.102.52.213:4340 , piece_id=GGQZRBZNC5DKXOZYV664XOFD2YUAZAL4RPRDQY7ZY6LUN6NKEFKQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.175372487 (16h3m3.958712586s) [noise=true , address=212.102.52.216:4405 , piece_id=CBCNKWI2VLVCUZDVM54E243ULDQUGYEH5EWRK2ES6V6LRUPWQIIA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.181671326 (16h3m3.965011425s) [noise=true , address=212.102.52.215:4376 , piece_id=RK5KEGAOHFZBU4NN7IO44XPBY5YAOANJ5VLPUU3XVBIGVVFSFZVA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.186238400 (16h3m3.969578509s) [noise=true , address=212.102.52.217:4418 , piece_id=EVBLYS5IQFKBL5TE3UTNM5SENH3COOX6W4JDF53SFFON3JOYQHMQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.190371015 (16h3m3.973711114s) [noise=true , address=212.102.52.213:4328 , piece_id=GUZPGD4HCE5PSNWBWZMLEDHA5AEMZZTEF7N2V4OXC4LCI7TIRIFQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.191589199 (16h3m3.974929298s) [noise=true , address=212.102.52.216:4408 , piece_id=IEQIPNTTQ3LN7LVVVQSSQU67NLWWG5DC6D5I5W33SKPKPN532DMQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.199054834 (16h3m3.982394933s) [noise=true , address=212.102.52.215:4388 , piece_id=DNXSHYL6IAMUWAUUVCH6WANAI3HSQIMVJYKSSG3YKLKNS4S6DGJQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.220275774 (16h3m4.003615873s) [noise=true , address=212.102.52.217:4426 , piece_id=CK4UNDBYQQCEVPTT5WGWHBWFH6CRZO3JO7RZL3RO5NOYDFX3DJNA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.220628726 (16h3m4.003968825s) [noise=true , address=212.102.52.215:4378 , piece_id=6456XMZDE4RFGHOTOUDM2SNQJDOCHHERTJKKQA6R6KU4EQCD74XA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.226546558 (16h3m4.009886657s) [noise=true , address=212.102.52.213:4321 , piece_id=QFBCH5MLCAIFPWMHO4NNQJLPIX7QG5LPQJUHG77WMPI6VPIBQIQA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.226564292 (16h3m4.009904421s) [seg_pos={0 0}, upload=15359]: Satisfied cohort requirements after 16 pieces
May 25 18:07:18.226741169 (16h3m4.010081268s) [noise=true , address=212.102.52.214:4368 , piece_id=OPJWV4TDPYNLLY2A5TSKGCUGX542YUX2OMPMILUTPE3TJR4L72AA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zavf82Jv8HWUWhuGf2nXnhiF1GDupLBVhu6urZQP6N6j71R3C): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.226748514 (16h3m4.010088613s) [noise=true , address=212.102.52.214:4368 , piece_id=OPJWV4TDPYNLLY2A5TSKGCUGX542YUX2OMPMILUTPE3TJR4L72AA, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.226822314 (16h3m4.010162413s) [noise=true , address=212.102.52.213:4343 , piece_id=JA6JX5GQOL6WT6MQV4G4UZ2M73QAW5ZII5OW5EO367VKYRESOGGA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.226831161 (16h3m4.01017126s) [noise=true , address=212.102.52.213:4343 , piece_id=JA6JX5GQOL6WT6MQV4G4UZ2M73QAW5ZII5OW5EO367VKYRESOGGA, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.226940960 (16h3m4.010281059s) [noise=true , address=212.102.52.217:4432 , piece_id=GZKMD3YNEMZNXRNHAPFBBXL6T7XOFUHKVJURDCJQ6CTCZRDUW4KQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CPW6Lyzfry6DmU86Fh3rW6u1RS3qpe79mYMaJRB5r4QEEQC6F): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.226946731 (16h3m4.01028683s) [noise=true , address=212.102.52.217:4432 , piece_id=GZKMD3YNEMZNXRNHAPFBBXL6T7XOFUHKVJURDCJQ6CTCZRDUW4KQ, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.227019280 (16h3m4.010359379s) [noise=true , address=212.102.52.216:4416 , piece_id=AMAJVMUOCVS5TOZLHMDOUSQ5YXCFSYEDT7BW4CW56M6VA6LIDJLA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.227026895 (16h3m4.010366994s) [noise=true , address=212.102.52.216:4416 , piece_id=AMAJVMUOCVS5TOZLHMDOUSQ5YXCFSYEDT7BW4CW56M6VA6LIDJLA, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.227043557 (16h3m4.010383656s) [seg_pos={0 0}, upload=15359]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 18:07:18.227113290 (16h3m4.010453389s) [upload=15359]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 18:07:18.227118159 (16h3m4.010458258s) [upload=15359]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 18:07:34.282635746 (16h3m20.065975866s) [upload=15359]: Done uploading segments...
May 25 18:07:34.282653911 (16h3m20.06599402s) [upload=15359]: Upload finished. err: <nil>
Duration: 4.301098315s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [..................................X................X........] duration:
23.224072073s trace:
https://jaeger.dev.storj.io/trace/467f79c1decb330
FAIL (2026-05-25 18:13:39 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
se=true , address=212.102.52.216:4416 , piece_id=AMAJVMUOCVS5TOZLHMDOUSQ5YXCFSYEDT7BW4CW56M6VA6LIDJLA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.813733143 (16h3m2.597073242s) [noise=true , address=212.102.52.213:4340 , piece_id=GGQZRBZNC5DKXOZYV664XOFD2YUAZAL4RPRDQY7ZY6LUN6NKEFKQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.814170728 (16h3m2.597510827s) [noise=true , address=212.102.52.214:4356 , piece_id=5OQLNMQN2NWE2OJO7FS44O6NI4NRMDCGYQQ3OTTH62CQDZZGHVQA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.814532990 (16h3m2.597873089s) [noise=true , address=212.102.52.216:4405 , piece_id=CBCNKWI2VLVCUZDVM54E243ULDQUGYEH5EWRK2ES6V6LRUPWQIIA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.814792664 (16h3m2.598132764s) [noise=true , address=212.102.52.214:4358 , piece_id=NDE6FEYH2OTGRTRIHGJBGJSV77ZUKHXAR4AON3NDKBU33EL4BGZQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.815026731 (16h3m2.59836683s) [noise=true , address=212.102.52.214:4347 , piece_id=6SXGEUYVLSOUSGPUY22UAXQKUWFMWVQOV3NHMMOOJY3NNL5XBAHA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.815255207 (16h3m2.598595296s) [noise=true , address=212.102.52.215:4369 , piece_id=TLNVSD3KWUSXOW3YLEMNRREQW2RMZABFJEEGX64XH7QYAXZBNDHQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.815526655 (16h3m2.598866754s) [noise=true , address=212.102.52.213:4343 , piece_id=JA6JX5GQOL6WT6MQV4G4UZ2M73QAW5ZII5OW5EO367VKYRESOGGA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.817053338 (16h3m2.600393437s) [noise=true , address=212.102.52.213:4328 , piece_id=GUZPGD4HCE5PSNWBWZMLEDHA5AEMZZTEF7N2V4OXC4LCI7TIRIFQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.817184408 (16h3m2.600524507s) [noise=true , address=212.102.52.215:4378 , piece_id=6456XMZDE4RFGHOTOUDM2SNQJDOCHHERTJKKQA6R6KU4EQCD74XA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.818090236 (16h3m2.601430335s) [noise=true , address=212.102.52.216:4408 , piece_id=IEQIPNTTQ3LN7LVVVQSSQU67NLWWG5DC6D5I5W33SKPKPN532DMQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.818269799 (16h3m2.601609898s) [noise=true , address=212.102.52.217:4418 , piece_id=EVBLYS5IQFKBL5TE3UTNM5SENH3COOX6W4JDF53SFFON3JOYQHMQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.819170618 (16h3m2.602510717s) [noise=true , address=212.102.52.217:4432 , piece_id=GZKMD3YNEMZNXRNHAPFBBXL6T7XOFUHKVJURDCJQ6CTCZRDUW4KQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.819559901 (16h3m2.6029s) [noise=true , address=212.102.52.217:4426 , piece_id=CK4UNDBYQQCEVPTT5WGWHBWFH6CRZO3JO7RZL3RO5NOYDFX3DJNA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.820329489 (16h3m2.603669588s) [noise=true , address=212.102.52.217:4431 , piece_id=3JXJF2YLDKTQDXDK666LSTYGYTMHSE3ZXJ7Y5WOUHAXWZVFRBRYA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.821320470 (16h3m2.604660559s) [noise=true , address=212.102.52.215:4388 , piece_id=DNXSHYL6IAMUWAUUVCH6WANAI3HSQIMVJYKSSG3YKLKNS4S6DGJQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:17.875060315 (16h3m3.658400414s) [upload=15359]: Next returned nil segment
May 25 18:07:17.875071437 (16h3m3.658411536s) [upload=15359]: Waiting for error group managing segments...
May 25 18:07:18.161378706 (16h3m3.944718805s) [noise=true , address=212.102.52.214:4356 , piece_id=5OQLNMQN2NWE2OJO7FS44O6NI4NRMDCGYQQ3OTTH62CQDZZGHVQA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.163883035 (16h3m3.947223134s) [noise=true , address=212.102.52.215:4369 , piece_id=TLNVSD3KWUSXOW3YLEMNRREQW2RMZABFJEEGX64XH7QYAXZBNDHQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.169179691 (16h3m3.9525198s) [noise=true , address=212.102.52.214:4347 , piece_id=6SXGEUYVLSOUSGPUY22UAXQKUWFMWVQOV3NHMMOOJY3NNL5XBAHA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.169953397 (16h3m3.953293496s) [noise=true , address=212.102.52.214:4358 , piece_id=NDE6FEYH2OTGRTRIHGJBGJSV77ZUKHXAR4AON3NDKBU33EL4BGZQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.170012670 (16h3m3.953352769s) [noise=true , address=212.102.52.216:4406 , piece_id=DQK2NNDQZWMM7KWBUMYVDJZTGT3SITN3M2ILQ7Y3QNIBV4WZXTTQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.173457273 (16h3m3.956797402s) [noise=true , address=212.102.52.217:4431 , piece_id=3JXJF2YLDKTQDXDK666LSTYGYTMHSE3ZXJ7Y5WOUHAXWZVFRBRYA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.173853839 (16h3m3.957193938s) [noise=true , address=212.102.52.213:4340 , piece_id=GGQZRBZNC5DKXOZYV664XOFD2YUAZAL4RPRDQY7ZY6LUN6NKEFKQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.175372487 (16h3m3.958712586s) [noise=true , address=212.102.52.216:4405 , piece_id=CBCNKWI2VLVCUZDVM54E243ULDQUGYEH5EWRK2ES6V6LRUPWQIIA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.181671326 (16h3m3.965011425s) [noise=true , address=212.102.52.215:4376 , piece_id=RK5KEGAOHFZBU4NN7IO44XPBY5YAOANJ5VLPUU3XVBIGVVFSFZVA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.186238400 (16h3m3.969578509s) [noise=true , address=212.102.52.217:4418 , piece_id=EVBLYS5IQFKBL5TE3UTNM5SENH3COOX6W4JDF53SFFON3JOYQHMQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.190371015 (16h3m3.973711114s) [noise=true , address=212.102.52.213:4328 , piece_id=GUZPGD4HCE5PSNWBWZMLEDHA5AEMZZTEF7N2V4OXC4LCI7TIRIFQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.191589199 (16h3m3.974929298s) [noise=true , address=212.102.52.216:4408 , piece_id=IEQIPNTTQ3LN7LVVVQSSQU67NLWWG5DC6D5I5W33SKPKPN532DMQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.199054834 (16h3m3.982394933s) [noise=true , address=212.102.52.215:4388 , piece_id=DNXSHYL6IAMUWAUUVCH6WANAI3HSQIMVJYKSSG3YKLKNS4S6DGJQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.220275774 (16h3m4.003615873s) [noise=true , address=212.102.52.217:4426 , piece_id=CK4UNDBYQQCEVPTT5WGWHBWFH6CRZO3JO7RZL3RO5NOYDFX3DJNA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.220628726 (16h3m4.003968825s) [noise=true , address=212.102.52.215:4378 , piece_id=6456XMZDE4RFGHOTOUDM2SNQJDOCHHERTJKKQA6R6KU4EQCD74XA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.226546558 (16h3m4.009886657s) [noise=true , address=212.102.52.213:4321 , piece_id=QFBCH5MLCAIFPWMHO4NNQJLPIX7QG5LPQJUHG77WMPI6VPIBQIQA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.226564292 (16h3m4.009904421s) [seg_pos={0 0}, upload=15359]: Satisfied cohort requirements after 16 pieces
May 25 18:07:18.226741169 (16h3m4.010081268s) [noise=true , address=212.102.52.214:4368 , piece_id=OPJWV4TDPYNLLY2A5TSKGCUGX542YUX2OMPMILUTPE3TJR4L72AA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zavf82Jv8HWUWhuGf2nXnhiF1GDupLBVhu6urZQP6N6j71R3C): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.226748514 (16h3m4.010088613s) [noise=true , address=212.102.52.214:4368 , piece_id=OPJWV4TDPYNLLY2A5TSKGCUGX542YUX2OMPMILUTPE3TJR4L72AA, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.226822314 (16h3m4.010162413s) [noise=true , address=212.102.52.213:4343 , piece_id=JA6JX5GQOL6WT6MQV4G4UZ2M73QAW5ZII5OW5EO367VKYRESOGGA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.226831161 (16h3m4.01017126s) [noise=true , address=212.102.52.213:4343 , piece_id=JA6JX5GQOL6WT6MQV4G4UZ2M73QAW5ZII5OW5EO367VKYRESOGGA, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.226940960 (16h3m4.010281059s) [noise=true , address=212.102.52.217:4432 , piece_id=GZKMD3YNEMZNXRNHAPFBBXL6T7XOFUHKVJURDCJQ6CTCZRDUW4KQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CPW6Lyzfry6DmU86Fh3rW6u1RS3qpe79mYMaJRB5r4QEEQC6F): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.226946731 (16h3m4.01028683s) [noise=true , address=212.102.52.217:4432 , piece_id=GZKMD3YNEMZNXRNHAPFBBXL6T7XOFUHKVJURDCJQ6CTCZRDUW4KQ, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.227019280 (16h3m4.010359379s) [noise=true , address=212.102.52.216:4416 , piece_id=AMAJVMUOCVS5TOZLHMDOUSQ5YXCFSYEDT7BW4CW56M6VA6LIDJLA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.227026895 (16h3m4.010366994s) [noise=true , address=212.102.52.216:4416 , piece_id=AMAJVMUOCVS5TOZLHMDOUSQ5YXCFSYEDT7BW4CW56M6VA6LIDJLA, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.227043557 (16h3m4.010383656s) [seg_pos={0 0}, upload=15359]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 18:07:18.227113290 (16h3m4.010453389s) [upload=15359]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 18:07:18.227118159 (16h3m4.010458258s) [upload=15359]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 18:07:34.282635746 (16h3m20.065975866s) [upload=15359]: Done uploading segments...
May 25 18:07:34.282653911 (16h3m20.06599402s) [upload=15359]: Upload finished. err: <nil>
Duration: 23.224083705s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.......................................X....................] duration:
3.680515514s trace:
https://jaeger.dev.storj.io/trace/14eb8131f680ed75
FAIL (2026-05-25 18:13:30 UTC)
orbiter warning: context deadline exceeded
Duration: 3.680535461s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
12 out of 60 is failed (threshold=3) [..............X....X.......X......X..XX.XX.X....X.....XX....] duration:
6.902019707s trace:
https://jaeger.dev.storj.io/trace/59b9e788be2903db
FAIL (2026-05-25 18:13:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.902045246s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)antiblip:
2 out of 60 is failed (threshold=3) [...................................X................X.......] duration:
41.771053346s trace:
https://jaeger.dev.storj.io/trace/467f79c1decb330
FAIL (2026-05-25 18:13:15 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
se=true , address=212.102.52.216:4416 , piece_id=AMAJVMUOCVS5TOZLHMDOUSQ5YXCFSYEDT7BW4CW56M6VA6LIDJLA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.813733143 (16h3m2.597073242s) [noise=true , address=212.102.52.213:4340 , piece_id=GGQZRBZNC5DKXOZYV664XOFD2YUAZAL4RPRDQY7ZY6LUN6NKEFKQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.814170728 (16h3m2.597510827s) [noise=true , address=212.102.52.214:4356 , piece_id=5OQLNMQN2NWE2OJO7FS44O6NI4NRMDCGYQQ3OTTH62CQDZZGHVQA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.814532990 (16h3m2.597873089s) [noise=true , address=212.102.52.216:4405 , piece_id=CBCNKWI2VLVCUZDVM54E243ULDQUGYEH5EWRK2ES6V6LRUPWQIIA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.814792664 (16h3m2.598132764s) [noise=true , address=212.102.52.214:4358 , piece_id=NDE6FEYH2OTGRTRIHGJBGJSV77ZUKHXAR4AON3NDKBU33EL4BGZQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.815026731 (16h3m2.59836683s) [noise=true , address=212.102.52.214:4347 , piece_id=6SXGEUYVLSOUSGPUY22UAXQKUWFMWVQOV3NHMMOOJY3NNL5XBAHA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.815255207 (16h3m2.598595296s) [noise=true , address=212.102.52.215:4369 , piece_id=TLNVSD3KWUSXOW3YLEMNRREQW2RMZABFJEEGX64XH7QYAXZBNDHQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.815526655 (16h3m2.598866754s) [noise=true , address=212.102.52.213:4343 , piece_id=JA6JX5GQOL6WT6MQV4G4UZ2M73QAW5ZII5OW5EO367VKYRESOGGA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.817053338 (16h3m2.600393437s) [noise=true , address=212.102.52.213:4328 , piece_id=GUZPGD4HCE5PSNWBWZMLEDHA5AEMZZTEF7N2V4OXC4LCI7TIRIFQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.817184408 (16h3m2.600524507s) [noise=true , address=212.102.52.215:4378 , piece_id=6456XMZDE4RFGHOTOUDM2SNQJDOCHHERTJKKQA6R6KU4EQCD74XA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.818090236 (16h3m2.601430335s) [noise=true , address=212.102.52.216:4408 , piece_id=IEQIPNTTQ3LN7LVVVQSSQU67NLWWG5DC6D5I5W33SKPKPN532DMQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.818269799 (16h3m2.601609898s) [noise=true , address=212.102.52.217:4418 , piece_id=EVBLYS5IQFKBL5TE3UTNM5SENH3COOX6W4JDF53SFFON3JOYQHMQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.819170618 (16h3m2.602510717s) [noise=true , address=212.102.52.217:4432 , piece_id=GZKMD3YNEMZNXRNHAPFBBXL6T7XOFUHKVJURDCJQ6CTCZRDUW4KQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.819559901 (16h3m2.6029s) [noise=true , address=212.102.52.217:4426 , piece_id=CK4UNDBYQQCEVPTT5WGWHBWFH6CRZO3JO7RZL3RO5NOYDFX3DJNA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.820329489 (16h3m2.603669588s) [noise=true , address=212.102.52.217:4431 , piece_id=3JXJF2YLDKTQDXDK666LSTYGYTMHSE3ZXJ7Y5WOUHAXWZVFRBRYA, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:16.821320470 (16h3m2.604660559s) [noise=true , address=212.102.52.215:4388 , piece_id=DNXSHYL6IAMUWAUUVCH6WANAI3HSQIMVJYKSSG3YKLKNS4S6DGJQ, seg_pos={0 0}, upload=15359]: Uploading piece...
May 25 18:07:17.875060315 (16h3m3.658400414s) [upload=15359]: Next returned nil segment
May 25 18:07:17.875071437 (16h3m3.658411536s) [upload=15359]: Waiting for error group managing segments...
May 25 18:07:18.161378706 (16h3m3.944718805s) [noise=true , address=212.102.52.214:4356 , piece_id=5OQLNMQN2NWE2OJO7FS44O6NI4NRMDCGYQQ3OTTH62CQDZZGHVQA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.163883035 (16h3m3.947223134s) [noise=true , address=212.102.52.215:4369 , piece_id=TLNVSD3KWUSXOW3YLEMNRREQW2RMZABFJEEGX64XH7QYAXZBNDHQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.169179691 (16h3m3.9525198s) [noise=true , address=212.102.52.214:4347 , piece_id=6SXGEUYVLSOUSGPUY22UAXQKUWFMWVQOV3NHMMOOJY3NNL5XBAHA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.169953397 (16h3m3.953293496s) [noise=true , address=212.102.52.214:4358 , piece_id=NDE6FEYH2OTGRTRIHGJBGJSV77ZUKHXAR4AON3NDKBU33EL4BGZQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.170012670 (16h3m3.953352769s) [noise=true , address=212.102.52.216:4406 , piece_id=DQK2NNDQZWMM7KWBUMYVDJZTGT3SITN3M2ILQ7Y3QNIBV4WZXTTQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.173457273 (16h3m3.956797402s) [noise=true , address=212.102.52.217:4431 , piece_id=3JXJF2YLDKTQDXDK666LSTYGYTMHSE3ZXJ7Y5WOUHAXWZVFRBRYA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.173853839 (16h3m3.957193938s) [noise=true , address=212.102.52.213:4340 , piece_id=GGQZRBZNC5DKXOZYV664XOFD2YUAZAL4RPRDQY7ZY6LUN6NKEFKQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.175372487 (16h3m3.958712586s) [noise=true , address=212.102.52.216:4405 , piece_id=CBCNKWI2VLVCUZDVM54E243ULDQUGYEH5EWRK2ES6V6LRUPWQIIA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.181671326 (16h3m3.965011425s) [noise=true , address=212.102.52.215:4376 , piece_id=RK5KEGAOHFZBU4NN7IO44XPBY5YAOANJ5VLPUU3XVBIGVVFSFZVA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.186238400 (16h3m3.969578509s) [noise=true , address=212.102.52.217:4418 , piece_id=EVBLYS5IQFKBL5TE3UTNM5SENH3COOX6W4JDF53SFFON3JOYQHMQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.190371015 (16h3m3.973711114s) [noise=true , address=212.102.52.213:4328 , piece_id=GUZPGD4HCE5PSNWBWZMLEDHA5AEMZZTEF7N2V4OXC4LCI7TIRIFQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.191589199 (16h3m3.974929298s) [noise=true , address=212.102.52.216:4408 , piece_id=IEQIPNTTQ3LN7LVVVQSSQU67NLWWG5DC6D5I5W33SKPKPN532DMQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.199054834 (16h3m3.982394933s) [noise=true , address=212.102.52.215:4388 , piece_id=DNXSHYL6IAMUWAUUVCH6WANAI3HSQIMVJYKSSG3YKLKNS4S6DGJQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.220275774 (16h3m4.003615873s) [noise=true , address=212.102.52.217:4426 , piece_id=CK4UNDBYQQCEVPTT5WGWHBWFH6CRZO3JO7RZL3RO5NOYDFX3DJNA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.220628726 (16h3m4.003968825s) [noise=true , address=212.102.52.215:4378 , piece_id=6456XMZDE4RFGHOTOUDM2SNQJDOCHHERTJKKQA6R6KU4EQCD74XA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.226546558 (16h3m4.009886657s) [noise=true , address=212.102.52.213:4321 , piece_id=QFBCH5MLCAIFPWMHO4NNQJLPIX7QG5LPQJUHG77WMPI6VPIBQIQA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: <nil>
May 25 18:07:18.226564292 (16h3m4.009904421s) [seg_pos={0 0}, upload=15359]: Satisfied cohort requirements after 16 pieces
May 25 18:07:18.226741169 (16h3m4.010081268s) [noise=true , address=212.102.52.214:4368 , piece_id=OPJWV4TDPYNLLY2A5TSKGCUGX542YUX2OMPMILUTPE3TJR4L72AA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zavf82Jv8HWUWhuGf2nXnhiF1GDupLBVhu6urZQP6N6j71R3C): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.226748514 (16h3m4.010088613s) [noise=true , address=212.102.52.214:4368 , piece_id=OPJWV4TDPYNLLY2A5TSKGCUGX542YUX2OMPMILUTPE3TJR4L72AA, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.226822314 (16h3m4.010162413s) [noise=true , address=212.102.52.213:4343 , piece_id=JA6JX5GQOL6WT6MQV4G4UZ2M73QAW5ZII5OW5EO367VKYRESOGGA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129SbAN2aTSXzo8JLELV2ssdkmT5k7kbTcvxSZE4sFsVaKw1Aok): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.226831161 (16h3m4.01017126s) [noise=true , address=212.102.52.213:4343 , piece_id=JA6JX5GQOL6WT6MQV4G4UZ2M73QAW5ZII5OW5EO367VKYRESOGGA, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.226940960 (16h3m4.010281059s) [noise=true , address=212.102.52.217:4432 , piece_id=GZKMD3YNEMZNXRNHAPFBBXL6T7XOFUHKVJURDCJQ6CTCZRDUW4KQ, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CPW6Lyzfry6DmU86Fh3rW6u1RS3qpe79mYMaJRB5r4QEEQC6F): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.226946731 (16h3m4.01028683s) [noise=true , address=212.102.52.217:4432 , piece_id=GZKMD3YNEMZNXRNHAPFBBXL6T7XOFUHKVJURDCJQ6CTCZRDUW4KQ, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.227019280 (16h3m4.010359379s) [noise=true , address=212.102.52.216:4416 , piece_id=AMAJVMUOCVS5TOZLHMDOUSQ5YXCFSYEDT7BW4CW56M6VA6LIDJLA, seg_pos={0 0}, upload=15359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Phe9CqWEgoJLbxBLeSUDWMMmah4JR5tuyPENLK3GrfKE3xBs): protocol: expected piece hash; context canceled; EOF
May 25 18:07:18.227026895 (16h3m4.010366994s) [noise=true , address=212.102.52.216:4416 , piece_id=AMAJVMUOCVS5TOZLHMDOUSQ5YXCFSYEDT7BW4CW56M6VA6LIDJLA, seg_pos={0 0}, upload=15359]: Long tail cancellation due to optimal threshold
May 25 18:07:18.227043557 (16h3m4.010383656s) [seg_pos={0 0}, upload=15359]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil>
May 25 18:07:18.227113290 (16h3m4.010453389s) [upload=15359]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
May 25 18:07:18.227118159 (16h3m4.010458258s) [upload=15359]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
May 25 18:07:34.282635746 (16h3m20.065975866s) [upload=15359]: Done uploading segments...
May 25 18:07:34.282653911 (16h3m20.06599402s) [upload=15359]: Upload finished. err: <nil>
Duration: 41.771091329s
core
uplink
uk-select-1
uk1-placement-17
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
682.609108ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/7e710747942acdb2
FAIL (2026-05-25 18:12:54 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 682.631451ms
core
satellite
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
12 out of 60 is failed (threshold=3) [...............X....X.......X......X..XX.XX.X....X.....XX...] duration:
7.049105586s trace:
https://jaeger.dev.storj.io/trace/59b9e788be2903db
FAIL (2026-05-25 18:12:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.049127487s
core
uplink
us-select-1
us1-placement-20
|