Orbiter History

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [X...............X..X........................................] duration: 10.105844041s trace: https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1 FAIL (2026-05-25 18:53:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 77]: Uploading piece... May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments... May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments... May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded Duration: 10.105867386s core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 3.815004475s trace: https://jaeger.dev.storj.io/trace/4d082fcde665b7a9 FAIL (2026-05-25 18:52:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.815021307s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [.X...............X..X.......................................] duration: 3.938808747s trace: https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1 FAIL (2026-05-25 18:52:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 77]: Uploading piece... May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments... May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments... May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded Duration: 3.938827733s core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [...XX......X...........X...........X...X...X................] duration: 6.591541806s trace: https://jaeger.dev.storj.io/trace/4fc3ac9cb273d912 FAIL (2026-05-25 18:52:19 UTC) orbiter warning: context deadline exceeded Duration: 6.591564471s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [..X...............X..X......................................] duration: 4.202645852s trace: https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1 FAIL (2026-05-25 18:52:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 77]: Uploading piece... May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments... May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments... May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded Duration: 4.202668805s core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 3.883744051s trace: https://jaeger.dev.storj.io/trace/4d082fcde665b7a9 FAIL (2026-05-25 18:51:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.883794658s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [...X...............X..X.....................................] duration: 5.109376845s trace: https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1 FAIL (2026-05-25 18:51:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 77]: Uploading piece... May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments... May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments... May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.10940011s core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [....XX......X...........X...........X...X...X...............] duration: 6.428583677s trace: https://jaeger.dev.storj.io/trace/4fc3ac9cb273d912 FAIL (2026-05-25 18:51:42 UTC) orbiter warning: context deadline exceeded Duration: 6.428608805s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 3.699441844s trace: https://jaeger.dev.storj.io/trace/4d082fcde665b7a9 FAIL (2026-05-25 18:51:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.699464286s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.....XX......X...........X...........X...X...X..............] duration: 18.217995953s trace: https://jaeger.dev.storj.io/trace/4fc3ac9cb273d912 FAIL (2026-05-25 18:51:01 UTC) orbiter warning: context deadline exceeded Duration: 18.218098379s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [....X...............X..X....................................] duration: 4.070407754s trace: https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1 FAIL (2026-05-25 18:50:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 77]: Uploading piece... May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments... May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments... May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded Duration: 4.070426881s core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [X.....XX......X...........X...........X...X...X.............] duration: 12.672030473s trace: https://jaeger.dev.storj.io/trace/4fc3ac9cb273d912 FAIL (2026-05-25 18:50:26 UTC) orbiter warning: context deadline exceeded Duration: 12.672058074s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 3.800634644s trace: https://jaeger.dev.storj.io/trace/4d082fcde665b7a9 FAIL (2026-05-25 18:50:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.800656175s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [.....X...............X..X...................................] duration: 3.969038956s trace: https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1 FAIL (2026-05-25 18:50:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 77]: Uploading piece... May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments... May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments... May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded Duration: 3.969052252s core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [......X...............X..X..................................] duration: 4.06806289s trace: https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1 FAIL (2026-05-25 18:49:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 77]: Uploading piece... May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments... May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments... May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded Duration: 4.068079353s core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 3.931038499s trace: https://jaeger.dev.storj.io/trace/4d082fcde665b7a9 FAIL (2026-05-25 18:49:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.93105974s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.X.....XX......X...........X...........X...X...X............] duration: 6.930422679s trace: https://jaeger.dev.storj.io/trace/4fc3ac9cb273d912 FAIL (2026-05-25 18:49:17 UTC) orbiter warning: context deadline exceeded Duration: 6.930436266s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [.......X...............X..X.................................] duration: 4.030520011s trace: https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1 FAIL (2026-05-25 18:48:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 77]: Uploading piece... May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments... May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments... May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded Duration: 4.030535279s core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 3.794137647s trace: https://jaeger.dev.storj.io/trace/4d082fcde665b7a9 FAIL (2026-05-25 18:48:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.794158837s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..X.....XX......X...........X...........X...X...X...........] duration: 11.966888627s trace: https://jaeger.dev.storj.io/trace/4fc3ac9cb273d912 FAIL (2026-05-25 18:48:38 UTC) orbiter warning: context deadline exceeded Duration: 11.96691183s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [........X...............X..X................................] duration: 4.319154442s trace: https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1 FAIL (2026-05-25 18:48:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 77]: Uploading piece... May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments... May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments... May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded Duration: 4.319172007s core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...X.....XX......X...........X...........X...X...X..........] duration: 7.002969293s trace: https://jaeger.dev.storj.io/trace/4fc3ac9cb273d912 FAIL (2026-05-25 18:47:59 UTC) orbiter warning: context deadline exceeded Duration: 7.002979342s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [.........X...............X..X...............................] duration: 4.022972222s trace: https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1 FAIL (2026-05-25 18:47:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 77]: Uploading piece... May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments... May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments... May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded Duration: 4.02299158s core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.973788868s trace: https://jaeger.dev.storj.io/trace/4d082fcde665b7a9 FAIL (2026-05-25 18:47:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.973815909s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [....X.....XX......X...........X...........X...X...X.........] duration: 17.160113155s trace: https://jaeger.dev.storj.io/trace/4fc3ac9cb273d912 FAIL (2026-05-25 18:47:02 UTC) orbiter warning: context deadline exceeded Duration: 17.160124325s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [X....X.....XX......X...........X...........X...X...X........] duration: 31.53351609s trace: https://jaeger.dev.storj.io/trace/4fc3ac9cb273d912 FAIL (2026-05-25 18:46:44 UTC) orbiter warning: context deadline exceeded Duration: 31.533535879s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.802786298s trace: https://jaeger.dev.storj.io/trace/4d082fcde665b7a9 FAIL (2026-05-25 18:46:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.802810856s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [..........X...............X..X..............................] duration: 4.062760314s trace: https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1 FAIL (2026-05-25 18:46:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 77]: Uploading piece... May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments... May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments... May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded Duration: 4.062793457s core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.X....X.....XX......X...........X...........X...X...X.......] duration: 6.553779748s trace: https://jaeger.dev.storj.io/trace/4fc3ac9cb273d912 FAIL (2026-05-25 18:46:01 UTC) orbiter warning: context deadline exceeded Duration: 6.553798254s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.692531177s trace: https://jaeger.dev.storj.io/trace/4d082fcde665b7a9 FAIL (2026-05-25 18:45:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.69256377s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [X.X....X.....XX......X...........X...........X...X...X......] duration: 16.339591643s trace: https://jaeger.dev.storj.io/trace/4fc3ac9cb273d912 FAIL (2026-05-25 18:45:41 UTC) orbiter warning: context deadline exceeded Duration: 16.339614949s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 4.440876403s trace: https://jaeger.dev.storj.io/trace/4d082fcde665b7a9 FAIL (2026-05-25 18:45:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.440919405s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [...........X...............X..X.............................] duration: 4.008307119s trace: https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1 FAIL (2026-05-25 18:45:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 77]: Uploading piece... May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments... May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments... May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded Duration: 4.008327558s core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.953560166s trace: https://jaeger.dev.storj.io/trace/4d082fcde665b7a9 FAIL (2026-05-25 18:45:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.953593258s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [XX.X....X.....XX......X...........X...........X...X...X.....] duration: 12.206987803s trace: https://jaeger.dev.storj.io/trace/4fc3ac9cb273d912 FAIL (2026-05-25 18:44:48 UTC) orbiter warning: context deadline exceeded Duration: 12.206999295s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.XX.X....X.....XX......X...........X...........X...X...X....] duration: 34.221929739s trace: https://jaeger.dev.storj.io/trace/4fc3ac9cb273d912 FAIL (2026-05-25 18:44:35 UTC) orbiter warning: context deadline exceeded Duration: 34.221951712s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 4.874847403s trace: https://jaeger.dev.storj.io/trace/4d082fcde665b7a9 FAIL (2026-05-25 18:44:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.87486704s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [............X...............X..X............................] duration: 4.171122162s trace: https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1 FAIL (2026-05-25 18:44:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 77]: Uploading piece... May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments... May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments... May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded Duration: 4.171144306s core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 4.056460191s trace: https://jaeger.dev.storj.io/trace/4d082fcde665b7a9 FAIL (2026-05-25 18:44:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.056504354s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [.............X...............X..X...........................] duration: 3.972743594s trace: https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1 FAIL (2026-05-25 18:43:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 77]: Uploading piece... May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments... May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments... May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded Duration: 3.972759014s core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [X.XX.X....X.....XX......X...........X...........X...X...X...] duration: 7.156560311s trace: https://jaeger.dev.storj.io/trace/4fc3ac9cb273d912 FAIL (2026-05-25 18:43:44 UTC) orbiter warning: context deadline exceeded Duration: 7.156608603s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [XX.XX.X....X.....XX......X...........X...........X...X...X..] duration: 8.682121627s trace: https://jaeger.dev.storj.io/trace/4fc3ac9cb273d912 FAIL (2026-05-25 18:43:26 UTC) orbiter warning: context deadline exceeded Duration: 8.682144049s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 4.030157002s trace: https://jaeger.dev.storj.io/trace/4d082fcde665b7a9 FAIL (2026-05-25 18:43:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.030179005s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [..............X...............X..X..........................] duration: 3.905652132s trace: https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1 FAIL (2026-05-25 18:43:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 77]: Uploading piece... May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments... May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments... May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded Duration: 3.905672031s core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.XX.XX.X....X.....XX......X...........X...........X...X...X.] duration: 30.602233073s trace: https://jaeger.dev.storj.io/trace/4fc3ac9cb273d912 FAIL (2026-05-25 18:43:01 UTC) orbiter warning: context deadline exceeded Duration: 30.602253331s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [...............X...............X..X.........................] duration: 5.602778442s trace: https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1 FAIL (2026-05-25 18:42:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 77]: Uploading piece... May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments... May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments... May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.60281416s core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [X.....................................................X.....] duration: 3.844545668s trace: https://jaeger.dev.storj.io/trace/4d082fcde665b7a9 FAIL (2026-05-25 18:42:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.844566036s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [..XX.XX.X....X.....XX......X...........X...........X...X...X] duration: 1m0.00098248s trace: https://jaeger.dev.storj.io/trace/4fc3ac9cb273d912 FAIL (2026-05-25 18:42:25 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001002459s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.X.....................................................X....] duration: 3.921412099s trace: https://jaeger.dev.storj.io/trace/4d082fcde665b7a9 FAIL (2026-05-25 18:42:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.921434082s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [................X...............X..X........................] duration: 4.141581023s trace: https://jaeger.dev.storj.io/trace/7a1964cec7dc06f1 FAIL (2026-05-25 18:41:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 77]: Uploading piece... May 25 18:20:50.088898735 (16h16m35.872238834s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089335718 (16h16m35.872675817s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.089738658 (16h16m35.873078747s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.090791806 (16h16m35.874131905s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091183283 (16h16m35.874523382s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091570923 (16h16m35.874911022s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.091882068 (16h16m35.875222157s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092229971 (16h16m35.87557007s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092535213 (16h16m35.875875312s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.092821670 (16h16m35.876161769s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093095122 (16h16m35.876435221s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093408419 (16h16m35.876748518s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.093697010 (16h16m35.877037099s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094006180 (16h16m35.877346269s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:50.094366327 (16h16m35.877706426s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Uploading piece... May 25 18:20:51.160092429 (16h16m36.943432528s) [upload=15577]: Next returned nil segment May 25 18:20:51.160103189 (16h16m36.943443289s) [upload=15577]: Waiting for error group managing segments... May 25 18:20:51.436752869 (16h16m37.220092968s) [noise=true , address=212.102.52.214:4349 , piece_id=DBHHZEZORY5EKZBXNPW6BVLBOERLSVWH6GAZI3PZP464KNW5IK4A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.442289854 (16h16m37.225629953s) [noise=true , address=212.102.52.214:4351 , piece_id=ZPCXDT73ZC7DNWXAPNRUO5I5X7BYE4XCFH6J7CTIC345L66MZW2Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.444314717 (16h16m37.227654816s) [noise=true , address=212.102.52.213:4340 , piece_id=TBYFRTU5DOYVVQCILUAXLSLGKMRK5E72BAXDVH42NEDNICPBTBJQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.445599629 (16h16m37.228939729s) [noise=true , address=212.102.52.213:4329 , piece_id=V3HGH6WYZRYGB44YVGP4XD6N3WVO6IXZMAMHXEB2ELIZTQVFY2NQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.447581912 (16h16m37.230922011s) [noise=true , address=212.102.52.215:4390 , piece_id=GMCYG3X4USPS6CN5LBBWNI72E4J6NANMNVBHHB6EU5VSNGN4ANPA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.449342581 (16h16m37.23268268s) [noise=true , address=212.102.52.213:4331 , piece_id=UOU2NUFCKKAPLSDQ5Z42QW6L6J26JDQNP4BHO6RVIGSUX5KFAXMA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452203890 (16h16m37.235543989s) [noise=true , address=212.102.52.217:4434 , piece_id=AVY3B2BLYQX2Y6TZKAT4EE3JKUMGXOOEB2OID5TFNLLRF3TAMXHA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.452335812 (16h16m37.235675901s) [noise=true , address=212.102.52.214:4352 , piece_id=CZEXVWWJWCU7PCK5SKECIR7454WEH2SEPSOG5REBPSCE54JKYOCQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.453015970 (16h16m37.236356069s) [noise=true , address=212.102.52.216:4415 , piece_id=YPZXUWZFCOKOOSOADH7E77L7VOUWIMETRPZA3UYKG5STTSDJHUGA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.454515381 (16h16m37.23785547s) [noise=true , address=212.102.52.217:4440 , piece_id=SPGQDY4LTBFXUSYYCBEZ7PQ7RX4XKJ3PV7QUUEGNFCHR5MBYB6JA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.455034050 (16h16m37.238374139s) [noise=true , address=212.102.52.217:4431 , piece_id=N7SFZMEZ32WJBLIMP6P65XGBHXKPMQKEDNAIGCWCYDDB5BIXSDKQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.458988927 (16h16m37.242329016s) [noise=true , address=212.102.52.215:4388 , piece_id=NP4T2SFOU75LVGCH3WTLIEXFT3JXMOU3CTD35UCMOAA6C5LU7AKA, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.463065876 (16h16m37.246405975s) [noise=true , address=212.102.52.216:4407 , piece_id=GD4J7IPAWBYUTLGDQUB4NKHBKGYWDSY2QCTWLJFUQ2QNUGENNJEQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.467219712 (16h16m37.250559811s) [noise=true , address=212.102.52.215:4371 , piece_id=DQYFDCO7THUTPBEPFDAVID3HCZRSYNMUE2KQMIH5SCHWDB6XT2FQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.468380286 (16h16m37.251720385s) [noise=true , address=212.102.52.216:4416 , piece_id=4ATUA7XKNNUEXT2MP6XIFLT2YOHVY3ZI3BP4JBSMR6PCTUDOBIDQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.471996817 (16h16m37.255336916s) [noise=true , address=212.102.52.217:4424 , piece_id=ZDOEQPYMSQFRT47B3FVE4H4RDBIS23TB2NUVNTHY3ZH4QXRKTZTQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: <nil> May 25 18:20:51.472006284 (16h16m37.255346373s) [seg_pos={0 0}, upload=15577]: Satisfied cohort requirements after 16 pieces May 25 18:20:51.472185516 (16h16m37.255525615s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNtJaVKa9A71zjFz4em41XBwKn6jkUpGNCw279Rd7QCfqPL8K): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472192570 (16h16m37.255532669s) [noise=true , address=212.102.52.213:4342 , piece_id=BBH62HWIL6PZXFMKJZKKSU6S5X3V2WYJTF4QQZEE2APRCTZTKP2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472198472 (16h16m37.255538581s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sH5EigMH2sodFcDBup4WLsKZ1jEvQXP9xRRV1HrCEvYCLBZHz): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472205916 (16h16m37.255546015s) [noise=true , address=212.102.52.216:4400 , piece_id=HB5UHZDAAHWHMTUUXI36ZXKIBT7ZULB4G66BHVJVJZPNS44TAG2A, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472316817 (16h16m37.255656906s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wAfNSuNAdVkfMjxBhTsxFpMrKVephQNoWmgtBu4fvUEmGrHtD): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472324592 (16h16m37.255664681s) [noise=true , address=212.102.52.215:4370 , piece_id=ILTJSKFAHMUCMELSCEJKW6SPTD2UTCVWFT5TCRSZFQFHX7ZYZI7Q, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472550994 (16h16m37.255891083s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BHBjpvr3Y4dyKi2KHFy7MpiPzh291UT6X1MMvQLB2ts4xpx6p): protocol: expected piece hash; context canceled; EOF May 25 18:20:51.472555211 (16h16m37.255895311s) [noise=true , address=212.102.52.214:4364 , piece_id=SILZOY3SW665BHP4LM7OEDRBUCAFDMCPCFUK7JH5WDNI5C2M3ECQ, seg_pos={0 0}, upload=15577]: Long tail cancellation due to optimal threshold May 25 18:20:51.472589939 (16h16m37.255930038s) [seg_pos={0 0}, upload=15577]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 25 18:20:51.472649642 (16h16m37.255989741s) [upload=15577]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 25 18:20:51.472654231 (16h16m37.25599433s) [upload=15577]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 25 18:22:06.057151580 (16h17m51.840491679s) [upload=15577]: Done uploading segments... May 25 18:22:06.057234688 (16h17m51.840574797s) [upload=15577]: Upload finished. err: metaclient: context deadline exceeded Duration: 4.141601782s core uplink uk-select-1 uk1-placement-17

Orbiter Check History

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)