Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 452378 reports (Page 394 of 9048)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.413289517s trace: https://jaeger.dev.storj.io/trace/7fd32fc43d58b31c FAIL (2026-01-11 15:40:35 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.413322569s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................X...X....XX...........................] duration: 5.911573092s trace: https://jaeger.dev.storj.io/trace/15f47b4548974807 FAIL (2026-01-11 15:40:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 11 15:20:44.499817824 (887h51m44.37090086s) [upload=776950]: Starting upload Jan 11 15:20:44.499886175 (887h51m44.37096921s) [upload=776950]: Done starting upload Jan 11 15:20:44.499998148 (887h51m44.371081183s) [upload=776950]: Uploading segments... Jan 11 15:20:44.500806667 (887h51m44.371889712s) [upload=776950]: Got next segment. Inline: false Jan 11 15:20:44.500861151 (887h51m44.371944186s) [upload=776950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 15:20:44.500866932 (887h51m44.371949987s) [upload=776950]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 15:20:44.748047886 (887h51m44.619130931s) [seg_pos={0 0}, upload=776950]: Begin upload segment... Jan 11 15:20:44.748112808 (887h51m44.619195843s) [seg_pos={0 0}, upload=776950]: Done begin upload segment. Jan 11 15:20:44.748136734 (887h51m44.619219779s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.748152184 (887h51m44.619235249s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.748307379 (887h51m44.619390414s) [noise=true , address=145.79.227.33:4869 , piece_id=CH2RKQPDTDMB4XC4GT6I22SLZWSGZYCAUBSP3EAGN66KNNKODEFA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.748774909 (887h51m44.619857944s) [noise=true , address=145.79.227.12:4765 , piece_id=JIZAWBTRKZKBBHLTXFURJOBJXLAHF34MYA6ZCJWDTZKEWTKHSA7Q, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.749107543 (887h51m44.620190578s) [noise=true , address=145.79.227.22:4509 , piece_id=SAASLZ536HTQD5SSEK6XCLZLQHBQMU7XS5634662DDLIOK4SP4IQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.749386073 (887h51m44.620469118s) [noise=true , address=77.90.31.13:4627 , piece_id=GT4K2RYPOKWXA62QTYDFUVGQC7MMRWXXKB5ZJ6YQIKGKT3SWZEUA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.749723326 (887h51m44.620806371s) [noise=true , address=145.79.227.17:4972 , piece_id=WWFXI2QSE25XAV3LIXK26YMZURW3HJ7PJWVCZPU6Q3MJ7ZU4AVHA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.750167912 (887h51m44.621250947s) [noise=true , address=145.79.227.41:4085 , piece_id=CB27PJDJOIHXV2JLVVOY2JXXDZ7HB6JQZTH6YGYNIVTZQMTP4CJQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.750465619 (887h51m44.621548664s) [noise=true , address=145.79.227.13:4804 , piece_id=Z3H3KFSARJMZDAIWXZ7X52ERD73PHQ4EQTBAF7TH4LX6VHPDDLEQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.750839451 (887h51m44.621922486s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.751162165 (887h51m44.62224521s) [noise=true , address=145.79.227.5:4223 , piece_id=6RMRBZNGRFTUMGRDOJANB5CIBYJRU4G5EHZOFSMDSX6UNIKINJHQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.751499277 (887h51m44.622582312s) [noise=true , address=145.79.227.18:4006 , piece_id=RDSUXZAJXHDSQGNZ4BSOIGZYK2DBAOS3OKCXD7EKUXGLVB7SXFVQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.751801051 (887h51m44.622884086s) [noise=true , address=145.79.227.11:4729 , piece_id=OUQFHEARVMKMD3NOSYNK3SGS65GHO2VVWI33TMDSGNZSVI6U2GGQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.752107515 (887h51m44.62319056s) [noise=true , address=45.91.116.13:4156 , piece_id=ZSQICLD2EZCVFCWHWBBPHLBJQJ5EAIYQG2MBQWACDPAX5RDPKXOQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.752505893 (887h51m44.623588938s) [noise=true , address=145.79.227.19:4079 , piece_id=QMKDT3RBPNHPJMUIAJAFPEAYT4YJ7ECHCSGZWDNSBMSSJR34NUFA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.753696261 (887h51m44.624779306s) [noise=true , address=145.79.227.40:4051 , piece_id=3S7ANT2SY7UNVUGT76H4SM3OIC5CY6JTTE2LR3WVOIYWA6IIWB3A, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:47.390523966 (887h51m47.261607001s) [upload=776950]: Next returned nil segment Jan 11 15:20:47.390543012 (887h51m47.261626057s) [upload=776950]: Waiting for error group managing segments... Jan 11 15:20:47.822599671 (887h51m47.693682706s) [noise=true , address=145.79.227.19:4079 , piece_id=QMKDT3RBPNHPJMUIAJAFPEAYT4YJ7ECHCSGZWDNSBMSSJR34NUFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:47.998445353 (887h51m47.869528388s) [noise=true , address=45.91.116.13:4156 , piece_id=ZSQICLD2EZCVFCWHWBBPHLBJQJ5EAIYQG2MBQWACDPAX5RDPKXOQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:48.230600154 (887h51m48.101683199s) [noise=true , address=77.90.31.13:4627 , piece_id=GT4K2RYPOKWXA62QTYDFUVGQC7MMRWXXKB5ZJ6YQIKGKT3SWZEUA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.569294804 (887h51m49.440377839s) [noise=true , address=145.79.227.13:4804 , piece_id=Z3H3KFSARJMZDAIWXZ7X52ERD73PHQ4EQTBAF7TH4LX6VHPDDLEQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.591086838 (887h51m49.462169873s) [noise=true , address=145.79.227.12:4765 , piece_id=JIZAWBTRKZKBBHLTXFURJOBJXLAHF34MYA6ZCJWDTZKEWTKHSA7Q, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.733493227 (887h51m49.604576272s) [noise=true , address=145.79.227.41:4085 , piece_id=CB27PJDJOIHXV2JLVVOY2JXXDZ7HB6JQZTH6YGYNIVTZQMTP4CJQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.794463392 (887h51m49.665546427s) [noise=true , address=145.79.227.17:4972 , piece_id=WWFXI2QSE25XAV3LIXK26YMZURW3HJ7PJWVCZPU6Q3MJ7ZU4AVHA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.967238577 (887h51m49.838321632s) [noise=true , address=145.79.227.18:4006 , piece_id=RDSUXZAJXHDSQGNZ4BSOIGZYK2DBAOS3OKCXD7EKUXGLVB7SXFVQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.989894235 (887h51m49.86097727s) [noise=true , address=145.79.227.22:4509 , piece_id=SAASLZ536HTQD5SSEK6XCLZLQHBQMU7XS5634662DDLIOK4SP4IQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:50.725655336 (887h51m50.596738371s) [noise=true , address=145.79.227.40:4051 , piece_id=3S7ANT2SY7UNVUGT76H4SM3OIC5CY6JTTE2LR3WVOIYWA6IIWB3A, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:53.061473450 (887h51m52.932556485s) [noise=true , address=145.79.227.5:4223 , piece_id=6RMRBZNGRFTUMGRDOJANB5CIBYJRU4G5EHZOFSMDSX6UNIKINJHQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:53.664200566 (887h51m53.535283601s) [noise=true , address=145.79.227.33:4869 , piece_id=CH2RKQPDTDMB4XC4GT6I22SLZWSGZYCAUBSP3EAGN66KNNKODEFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:54.439611625 (887h51m54.31069466s) [noise=true , address=145.79.227.11:4729 , piece_id=OUQFHEARVMKMD3NOSYNK3SGS65GHO2VVWI33TMDSGNZSVI6U2GGQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:54.439630861 (887h51m54.310713896s) [seg_pos={0 0}, upload=776950]: Segment reached optimal threshold of 13 pieces. Jan 11 15:20:54.439937875 (887h51m54.311020911s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF Jan 11 15:20:54.439980266 (887h51m54.311063311s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold Jan 11 15:20:54.439998441 (887h51m54.311081476s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: read tcp 188.166.225.170:41126->145.79.227.49:4337: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 15:20:54.440006246 (887h51m54.311089281s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold Jan 11 15:20:54.440191488 (887h51m54.311274523s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1): protocol: read tcp 188.166.225.170:36086->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 15:20:54.440199785 (887h51m54.31128282s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold Jan 11 15:20:54.440216286 (887h51m54.311299321s) [seg_pos={0 0}, upload=776950]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 15:20:54.440321306 (887h51m54.311404341s) [upload=776950]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 15:20:54.440326495 (887h51m54.31140953s) [upload=776950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 15:20:54.726699085 (887h51m54.59778212s) [upload=776950]: Done uploading segments... Jan 11 15:20:54.726719413 (887h51m54.597802458s) [upload=776950]: Upload finished. err: <nil> Duration: 5.911601084s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 1.995371534s trace: https://jaeger.dev.storj.io/trace/c31ec8b6bb524c FAIL (2026-01-11 15:40:18 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.995403496s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................X...X....XX..........................] duration: 5.77452556s trace: https://jaeger.dev.storj.io/trace/15f47b4548974807 FAIL (2026-01-11 15:40:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 11 15:20:44.499817824 (887h51m44.37090086s) [upload=776950]: Starting upload Jan 11 15:20:44.499886175 (887h51m44.37096921s) [upload=776950]: Done starting upload Jan 11 15:20:44.499998148 (887h51m44.371081183s) [upload=776950]: Uploading segments... Jan 11 15:20:44.500806667 (887h51m44.371889712s) [upload=776950]: Got next segment. Inline: false Jan 11 15:20:44.500861151 (887h51m44.371944186s) [upload=776950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 15:20:44.500866932 (887h51m44.371949987s) [upload=776950]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 15:20:44.748047886 (887h51m44.619130931s) [seg_pos={0 0}, upload=776950]: Begin upload segment... Jan 11 15:20:44.748112808 (887h51m44.619195843s) [seg_pos={0 0}, upload=776950]: Done begin upload segment. Jan 11 15:20:44.748136734 (887h51m44.619219779s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.748152184 (887h51m44.619235249s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.748307379 (887h51m44.619390414s) [noise=true , address=145.79.227.33:4869 , piece_id=CH2RKQPDTDMB4XC4GT6I22SLZWSGZYCAUBSP3EAGN66KNNKODEFA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.748774909 (887h51m44.619857944s) [noise=true , address=145.79.227.12:4765 , piece_id=JIZAWBTRKZKBBHLTXFURJOBJXLAHF34MYA6ZCJWDTZKEWTKHSA7Q, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.749107543 (887h51m44.620190578s) [noise=true , address=145.79.227.22:4509 , piece_id=SAASLZ536HTQD5SSEK6XCLZLQHBQMU7XS5634662DDLIOK4SP4IQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.749386073 (887h51m44.620469118s) [noise=true , address=77.90.31.13:4627 , piece_id=GT4K2RYPOKWXA62QTYDFUVGQC7MMRWXXKB5ZJ6YQIKGKT3SWZEUA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.749723326 (887h51m44.620806371s) [noise=true , address=145.79.227.17:4972 , piece_id=WWFXI2QSE25XAV3LIXK26YMZURW3HJ7PJWVCZPU6Q3MJ7ZU4AVHA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.750167912 (887h51m44.621250947s) [noise=true , address=145.79.227.41:4085 , piece_id=CB27PJDJOIHXV2JLVVOY2JXXDZ7HB6JQZTH6YGYNIVTZQMTP4CJQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.750465619 (887h51m44.621548664s) [noise=true , address=145.79.227.13:4804 , piece_id=Z3H3KFSARJMZDAIWXZ7X52ERD73PHQ4EQTBAF7TH4LX6VHPDDLEQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.750839451 (887h51m44.621922486s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.751162165 (887h51m44.62224521s) [noise=true , address=145.79.227.5:4223 , piece_id=6RMRBZNGRFTUMGRDOJANB5CIBYJRU4G5EHZOFSMDSX6UNIKINJHQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.751499277 (887h51m44.622582312s) [noise=true , address=145.79.227.18:4006 , piece_id=RDSUXZAJXHDSQGNZ4BSOIGZYK2DBAOS3OKCXD7EKUXGLVB7SXFVQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.751801051 (887h51m44.622884086s) [noise=true , address=145.79.227.11:4729 , piece_id=OUQFHEARVMKMD3NOSYNK3SGS65GHO2VVWI33TMDSGNZSVI6U2GGQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.752107515 (887h51m44.62319056s) [noise=true , address=45.91.116.13:4156 , piece_id=ZSQICLD2EZCVFCWHWBBPHLBJQJ5EAIYQG2MBQWACDPAX5RDPKXOQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.752505893 (887h51m44.623588938s) [noise=true , address=145.79.227.19:4079 , piece_id=QMKDT3RBPNHPJMUIAJAFPEAYT4YJ7ECHCSGZWDNSBMSSJR34NUFA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.753696261 (887h51m44.624779306s) [noise=true , address=145.79.227.40:4051 , piece_id=3S7ANT2SY7UNVUGT76H4SM3OIC5CY6JTTE2LR3WVOIYWA6IIWB3A, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:47.390523966 (887h51m47.261607001s) [upload=776950]: Next returned nil segment Jan 11 15:20:47.390543012 (887h51m47.261626057s) [upload=776950]: Waiting for error group managing segments... Jan 11 15:20:47.822599671 (887h51m47.693682706s) [noise=true , address=145.79.227.19:4079 , piece_id=QMKDT3RBPNHPJMUIAJAFPEAYT4YJ7ECHCSGZWDNSBMSSJR34NUFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:47.998445353 (887h51m47.869528388s) [noise=true , address=45.91.116.13:4156 , piece_id=ZSQICLD2EZCVFCWHWBBPHLBJQJ5EAIYQG2MBQWACDPAX5RDPKXOQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:48.230600154 (887h51m48.101683199s) [noise=true , address=77.90.31.13:4627 , piece_id=GT4K2RYPOKWXA62QTYDFUVGQC7MMRWXXKB5ZJ6YQIKGKT3SWZEUA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.569294804 (887h51m49.440377839s) [noise=true , address=145.79.227.13:4804 , piece_id=Z3H3KFSARJMZDAIWXZ7X52ERD73PHQ4EQTBAF7TH4LX6VHPDDLEQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.591086838 (887h51m49.462169873s) [noise=true , address=145.79.227.12:4765 , piece_id=JIZAWBTRKZKBBHLTXFURJOBJXLAHF34MYA6ZCJWDTZKEWTKHSA7Q, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.733493227 (887h51m49.604576272s) [noise=true , address=145.79.227.41:4085 , piece_id=CB27PJDJOIHXV2JLVVOY2JXXDZ7HB6JQZTH6YGYNIVTZQMTP4CJQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.794463392 (887h51m49.665546427s) [noise=true , address=145.79.227.17:4972 , piece_id=WWFXI2QSE25XAV3LIXK26YMZURW3HJ7PJWVCZPU6Q3MJ7ZU4AVHA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.967238577 (887h51m49.838321632s) [noise=true , address=145.79.227.18:4006 , piece_id=RDSUXZAJXHDSQGNZ4BSOIGZYK2DBAOS3OKCXD7EKUXGLVB7SXFVQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.989894235 (887h51m49.86097727s) [noise=true , address=145.79.227.22:4509 , piece_id=SAASLZ536HTQD5SSEK6XCLZLQHBQMU7XS5634662DDLIOK4SP4IQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:50.725655336 (887h51m50.596738371s) [noise=true , address=145.79.227.40:4051 , piece_id=3S7ANT2SY7UNVUGT76H4SM3OIC5CY6JTTE2LR3WVOIYWA6IIWB3A, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:53.061473450 (887h51m52.932556485s) [noise=true , address=145.79.227.5:4223 , piece_id=6RMRBZNGRFTUMGRDOJANB5CIBYJRU4G5EHZOFSMDSX6UNIKINJHQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:53.664200566 (887h51m53.535283601s) [noise=true , address=145.79.227.33:4869 , piece_id=CH2RKQPDTDMB4XC4GT6I22SLZWSGZYCAUBSP3EAGN66KNNKODEFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:54.439611625 (887h51m54.31069466s) [noise=true , address=145.79.227.11:4729 , piece_id=OUQFHEARVMKMD3NOSYNK3SGS65GHO2VVWI33TMDSGNZSVI6U2GGQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:54.439630861 (887h51m54.310713896s) [seg_pos={0 0}, upload=776950]: Segment reached optimal threshold of 13 pieces. Jan 11 15:20:54.439937875 (887h51m54.311020911s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF Jan 11 15:20:54.439980266 (887h51m54.311063311s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold Jan 11 15:20:54.439998441 (887h51m54.311081476s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: read tcp 188.166.225.170:41126->145.79.227.49:4337: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 15:20:54.440006246 (887h51m54.311089281s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold Jan 11 15:20:54.440191488 (887h51m54.311274523s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1): protocol: read tcp 188.166.225.170:36086->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 15:20:54.440199785 (887h51m54.31128282s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold Jan 11 15:20:54.440216286 (887h51m54.311299321s) [seg_pos={0 0}, upload=776950]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 15:20:54.440321306 (887h51m54.311404341s) [upload=776950]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 15:20:54.440326495 (887h51m54.31140953s) [upload=776950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 15:20:54.726699085 (887h51m54.59778212s) [upload=776950]: Done uploading segments... Jan 11 15:20:54.726719413 (887h51m54.597802458s) [upload=776950]: Upload finished. err: <nil> Duration: 5.774613338s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [XX.X..X......X.X.....................X........X......X.X....] duration: 33.001506568s trace: https://jaeger.dev.storj.io/trace/627dde93e011c19f FAIL (2026-01-11 15:40:06 UTC) orbiter warning: context deadline exceeded Duration: 33.001533578s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [XXXXX.XXXXXXXXX.X.XX.XX...X..XXX......XXXXX...X.XXXXXXXXXXXX] duration: 30.001204382s trace: https://jaeger.dev.storj.io/trace/3a62cbec14b17644 FAIL (2026-01-11 15:40:04 UTC) orbiter warning: context deadline exceeded Duration: 30.001237457s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [X.XXXXXXXXXXXX.XXXXX.XXXXXXXXXX.XX.XXXXX.XXXXX.X.XXX.XXX.XXX] duration: 5.00207562s trace: https://jaeger.dev.storj.io/trace/5881471c60e58436 FAIL (2026-01-11 15:40:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002097883s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.498793902s trace: https://jaeger.dev.storj.io/trace/7fd32fc43d58b31c FAIL (2026-01-11 15:39:33 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.498812618s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [XXXXXX.XXXXXXXXX.X.XX.XX...X..XXX......XXXXX...X.XXXXXXXXXXX] duration: 30.001286801s trace: https://jaeger.dev.storj.io/trace/20a88d2f533d7994 FAIL (2026-01-11 15:39:29 UTC) orbiter warning: context deadline exceeded Duration: 30.001308291s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.599580752s trace: https://jaeger.dev.storj.io/trace/db08c1a98355252 FAIL (2026-01-11 15:39:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, Piece ID: D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA) context deadline exceeded; piecestore: (Node ID: 1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, Piece ID: IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ) context deadline exceeded; piecestore: (Node ID: 12iDqXUpUqsDwhzKGNY1kMqEhS6vbHiESooVR9xu92B3SGybNmn, Piece ID: 3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ) context deadline exceeded ======LOGS====== =ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.515238933 (887h51m9.386321968s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.515935319 (887h51m9.387018354s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.516392410 (887h51m9.387475455s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.516885660 (887h51m9.387968705s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.517268609 (887h51m9.388351654s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.517740817 (887h51m9.388823852s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.518341942 (887h51m9.389424977s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.518779846 (887h51m9.389862881s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.519286630 (887h51m9.390369676s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.521517579 (887h51m9.392600614s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.524918765 (887h51m9.39600181s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.527183728 (887h51m9.398266783s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.529232859 (887h51m9.400315894s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.532984954 (887h51m9.404067999s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.536582354 (887h51m9.407665399s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.539559423 (887h51m9.410642468s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:10.837257700 (887h51m10.708340735s) [upload=776945]: Next returned nil segment Jan 11 15:20:10.837276767 (887h51m10.708359812s) [upload=776945]: Waiting for error group managing segments... Jan 11 15:20:11.321612300 (887h51m11.192695346s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:12.321151387 (887h51m12.192234433s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:12.481216511 (887h51m12.352299546s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:12.632251410 (887h51m12.503334445s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.302406869 (887h51m13.173489904s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.417486975 (887h51m13.28857001s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.437898801 (887h51m13.308981846s) [noise=true , address=145.79.227.33:4897 , piece_id=ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.666780260 (887h51m13.537863305s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.677746121 (887h51m13.548829156s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:14.339206281 (887h51m14.210289316s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:14.557434685 (887h51m14.42851773s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:15.047052989 (887h51m14.918136024s) [noise=true , address=77.90.31.11:4574 , piece_id=D6H557N23KO5L32TEYGYNSY7YHRPTW4VSN56ZLQYYAI5R6RN3FSA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:15.347734503 (887h51m15.218817548s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:15.431090403 (887h51m15.302173448s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:15.431113807 (887h51m15.302196852s) [seg_pos={0 0}, upload=776945]: Segment reached optimal threshold of 14 pieces. Jan 11 15:20:15.431355577 (887h51m15.302438622s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cCPRmsdkrVKFyMfLdBNYhnDhcA2Los7kbvuYUT6dcofPLVXZ7): protocol: expected piece hash; context canceled; EOF Jan 11 15:20:15.431362331 (887h51m15.302445376s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold Jan 11 15:20:15.431472140 (887h51m15.302555175s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Jan 11 15:20:15.431491286 (887h51m15.302574321s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold Jan 11 15:20:15.431505644 (887h51m15.302588679s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9): protocol: expected piece hash; context canceled; EOF Jan 11 15:20:15.431513218 (887h51m15.302596263s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold Jan 11 15:20:15.431640601 (887h51m15.302723646s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: read tcp 188.166.225.170:45042->145.79.227.30:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 15:20:15.431654497 (887h51m15.302737572s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold Jan 11 15:20:15.431749248 (887h51m15.302832283s) [seg_pos={0 0}, upload=776945]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 11 15:20:15.431840110 (887h51m15.302923155s) [upload=776945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 15:20:15.431847053 (887h51m15.302930088s) [upload=776945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 15:20:15.728771354 (887h51m15.599854419s) [upload=776945]: Done uploading segments... Jan 11 15:20:15.728795108 (887h51m15.599878153s) [upload=776945]: Upload finished. err: <nil> Duration: 5.599609458s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 1.960779607s trace: https://jaeger.dev.storj.io/trace/c31ec8b6bb524c FAIL (2026-01-11 15:39:19 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.960806809s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [.X.XXXXXXXXXXXX.XXXXX.XXXXXXXXXX.XX.XXXXX.XXXXX.X.XXX.XXX.XX] duration: 5.000885907s trace: https://jaeger.dev.storj.io/trace/2cbaec69ba740d1c FAIL (2026-01-11 15:39:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000909934s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................X...X....XX.........................] duration: 5.796674517s trace: https://jaeger.dev.storj.io/trace/15f47b4548974807 FAIL (2026-01-11 15:39:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 11 15:20:44.499817824 (887h51m44.37090086s) [upload=776950]: Starting upload Jan 11 15:20:44.499886175 (887h51m44.37096921s) [upload=776950]: Done starting upload Jan 11 15:20:44.499998148 (887h51m44.371081183s) [upload=776950]: Uploading segments... Jan 11 15:20:44.500806667 (887h51m44.371889712s) [upload=776950]: Got next segment. Inline: false Jan 11 15:20:44.500861151 (887h51m44.371944186s) [upload=776950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 15:20:44.500866932 (887h51m44.371949987s) [upload=776950]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 15:20:44.748047886 (887h51m44.619130931s) [seg_pos={0 0}, upload=776950]: Begin upload segment... Jan 11 15:20:44.748112808 (887h51m44.619195843s) [seg_pos={0 0}, upload=776950]: Done begin upload segment. Jan 11 15:20:44.748136734 (887h51m44.619219779s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.748152184 (887h51m44.619235249s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.748307379 (887h51m44.619390414s) [noise=true , address=145.79.227.33:4869 , piece_id=CH2RKQPDTDMB4XC4GT6I22SLZWSGZYCAUBSP3EAGN66KNNKODEFA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.748774909 (887h51m44.619857944s) [noise=true , address=145.79.227.12:4765 , piece_id=JIZAWBTRKZKBBHLTXFURJOBJXLAHF34MYA6ZCJWDTZKEWTKHSA7Q, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.749107543 (887h51m44.620190578s) [noise=true , address=145.79.227.22:4509 , piece_id=SAASLZ536HTQD5SSEK6XCLZLQHBQMU7XS5634662DDLIOK4SP4IQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.749386073 (887h51m44.620469118s) [noise=true , address=77.90.31.13:4627 , piece_id=GT4K2RYPOKWXA62QTYDFUVGQC7MMRWXXKB5ZJ6YQIKGKT3SWZEUA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.749723326 (887h51m44.620806371s) [noise=true , address=145.79.227.17:4972 , piece_id=WWFXI2QSE25XAV3LIXK26YMZURW3HJ7PJWVCZPU6Q3MJ7ZU4AVHA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.750167912 (887h51m44.621250947s) [noise=true , address=145.79.227.41:4085 , piece_id=CB27PJDJOIHXV2JLVVOY2JXXDZ7HB6JQZTH6YGYNIVTZQMTP4CJQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.750465619 (887h51m44.621548664s) [noise=true , address=145.79.227.13:4804 , piece_id=Z3H3KFSARJMZDAIWXZ7X52ERD73PHQ4EQTBAF7TH4LX6VHPDDLEQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.750839451 (887h51m44.621922486s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.751162165 (887h51m44.62224521s) [noise=true , address=145.79.227.5:4223 , piece_id=6RMRBZNGRFTUMGRDOJANB5CIBYJRU4G5EHZOFSMDSX6UNIKINJHQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.751499277 (887h51m44.622582312s) [noise=true , address=145.79.227.18:4006 , piece_id=RDSUXZAJXHDSQGNZ4BSOIGZYK2DBAOS3OKCXD7EKUXGLVB7SXFVQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.751801051 (887h51m44.622884086s) [noise=true , address=145.79.227.11:4729 , piece_id=OUQFHEARVMKMD3NOSYNK3SGS65GHO2VVWI33TMDSGNZSVI6U2GGQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.752107515 (887h51m44.62319056s) [noise=true , address=45.91.116.13:4156 , piece_id=ZSQICLD2EZCVFCWHWBBPHLBJQJ5EAIYQG2MBQWACDPAX5RDPKXOQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.752505893 (887h51m44.623588938s) [noise=true , address=145.79.227.19:4079 , piece_id=QMKDT3RBPNHPJMUIAJAFPEAYT4YJ7ECHCSGZWDNSBMSSJR34NUFA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.753696261 (887h51m44.624779306s) [noise=true , address=145.79.227.40:4051 , piece_id=3S7ANT2SY7UNVUGT76H4SM3OIC5CY6JTTE2LR3WVOIYWA6IIWB3A, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:47.390523966 (887h51m47.261607001s) [upload=776950]: Next returned nil segment Jan 11 15:20:47.390543012 (887h51m47.261626057s) [upload=776950]: Waiting for error group managing segments... Jan 11 15:20:47.822599671 (887h51m47.693682706s) [noise=true , address=145.79.227.19:4079 , piece_id=QMKDT3RBPNHPJMUIAJAFPEAYT4YJ7ECHCSGZWDNSBMSSJR34NUFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:47.998445353 (887h51m47.869528388s) [noise=true , address=45.91.116.13:4156 , piece_id=ZSQICLD2EZCVFCWHWBBPHLBJQJ5EAIYQG2MBQWACDPAX5RDPKXOQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:48.230600154 (887h51m48.101683199s) [noise=true , address=77.90.31.13:4627 , piece_id=GT4K2RYPOKWXA62QTYDFUVGQC7MMRWXXKB5ZJ6YQIKGKT3SWZEUA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.569294804 (887h51m49.440377839s) [noise=true , address=145.79.227.13:4804 , piece_id=Z3H3KFSARJMZDAIWXZ7X52ERD73PHQ4EQTBAF7TH4LX6VHPDDLEQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.591086838 (887h51m49.462169873s) [noise=true , address=145.79.227.12:4765 , piece_id=JIZAWBTRKZKBBHLTXFURJOBJXLAHF34MYA6ZCJWDTZKEWTKHSA7Q, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.733493227 (887h51m49.604576272s) [noise=true , address=145.79.227.41:4085 , piece_id=CB27PJDJOIHXV2JLVVOY2JXXDZ7HB6JQZTH6YGYNIVTZQMTP4CJQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.794463392 (887h51m49.665546427s) [noise=true , address=145.79.227.17:4972 , piece_id=WWFXI2QSE25XAV3LIXK26YMZURW3HJ7PJWVCZPU6Q3MJ7ZU4AVHA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.967238577 (887h51m49.838321632s) [noise=true , address=145.79.227.18:4006 , piece_id=RDSUXZAJXHDSQGNZ4BSOIGZYK2DBAOS3OKCXD7EKUXGLVB7SXFVQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.989894235 (887h51m49.86097727s) [noise=true , address=145.79.227.22:4509 , piece_id=SAASLZ536HTQD5SSEK6XCLZLQHBQMU7XS5634662DDLIOK4SP4IQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:50.725655336 (887h51m50.596738371s) [noise=true , address=145.79.227.40:4051 , piece_id=3S7ANT2SY7UNVUGT76H4SM3OIC5CY6JTTE2LR3WVOIYWA6IIWB3A, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:53.061473450 (887h51m52.932556485s) [noise=true , address=145.79.227.5:4223 , piece_id=6RMRBZNGRFTUMGRDOJANB5CIBYJRU4G5EHZOFSMDSX6UNIKINJHQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:53.664200566 (887h51m53.535283601s) [noise=true , address=145.79.227.33:4869 , piece_id=CH2RKQPDTDMB4XC4GT6I22SLZWSGZYCAUBSP3EAGN66KNNKODEFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:54.439611625 (887h51m54.31069466s) [noise=true , address=145.79.227.11:4729 , piece_id=OUQFHEARVMKMD3NOSYNK3SGS65GHO2VVWI33TMDSGNZSVI6U2GGQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:54.439630861 (887h51m54.310713896s) [seg_pos={0 0}, upload=776950]: Segment reached optimal threshold of 13 pieces. Jan 11 15:20:54.439937875 (887h51m54.311020911s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF Jan 11 15:20:54.439980266 (887h51m54.311063311s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold Jan 11 15:20:54.439998441 (887h51m54.311081476s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: read tcp 188.166.225.170:41126->145.79.227.49:4337: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 15:20:54.440006246 (887h51m54.311089281s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold Jan 11 15:20:54.440191488 (887h51m54.311274523s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1): protocol: read tcp 188.166.225.170:36086->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 15:20:54.440199785 (887h51m54.31128282s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold Jan 11 15:20:54.440216286 (887h51m54.311299321s) [seg_pos={0 0}, upload=776950]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 15:20:54.440321306 (887h51m54.311404341s) [upload=776950]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 15:20:54.440326495 (887h51m54.31140953s) [upload=776950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 15:20:54.726699085 (887h51m54.59778212s) [upload=776950]: Done uploading segments... Jan 11 15:20:54.726719413 (887h51m54.597802458s) [upload=776950]: Upload finished. err: <nil> Duration: 5.796706839s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [X.X.XXXXXXXXXXXX.XXXXX.XXXXXXXXXX.XX.XXXXX.XXXXX.X.XXX.XXX.X] duration: 5.001495799s trace: https://jaeger.dev.storj.io/trace/7966659577a37b70 FAIL (2026-01-11 15:38:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001546835s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................X...X....XX........................] duration: 5.869221037s trace: https://jaeger.dev.storj.io/trace/15f47b4548974807 FAIL (2026-01-11 15:38:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 11 15:20:44.499817824 (887h51m44.37090086s) [upload=776950]: Starting upload Jan 11 15:20:44.499886175 (887h51m44.37096921s) [upload=776950]: Done starting upload Jan 11 15:20:44.499998148 (887h51m44.371081183s) [upload=776950]: Uploading segments... Jan 11 15:20:44.500806667 (887h51m44.371889712s) [upload=776950]: Got next segment. Inline: false Jan 11 15:20:44.500861151 (887h51m44.371944186s) [upload=776950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 15:20:44.500866932 (887h51m44.371949987s) [upload=776950]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 15:20:44.748047886 (887h51m44.619130931s) [seg_pos={0 0}, upload=776950]: Begin upload segment... Jan 11 15:20:44.748112808 (887h51m44.619195843s) [seg_pos={0 0}, upload=776950]: Done begin upload segment. Jan 11 15:20:44.748136734 (887h51m44.619219779s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.748152184 (887h51m44.619235249s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.748307379 (887h51m44.619390414s) [noise=true , address=145.79.227.33:4869 , piece_id=CH2RKQPDTDMB4XC4GT6I22SLZWSGZYCAUBSP3EAGN66KNNKODEFA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.748774909 (887h51m44.619857944s) [noise=true , address=145.79.227.12:4765 , piece_id=JIZAWBTRKZKBBHLTXFURJOBJXLAHF34MYA6ZCJWDTZKEWTKHSA7Q, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.749107543 (887h51m44.620190578s) [noise=true , address=145.79.227.22:4509 , piece_id=SAASLZ536HTQD5SSEK6XCLZLQHBQMU7XS5634662DDLIOK4SP4IQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.749386073 (887h51m44.620469118s) [noise=true , address=77.90.31.13:4627 , piece_id=GT4K2RYPOKWXA62QTYDFUVGQC7MMRWXXKB5ZJ6YQIKGKT3SWZEUA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.749723326 (887h51m44.620806371s) [noise=true , address=145.79.227.17:4972 , piece_id=WWFXI2QSE25XAV3LIXK26YMZURW3HJ7PJWVCZPU6Q3MJ7ZU4AVHA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.750167912 (887h51m44.621250947s) [noise=true , address=145.79.227.41:4085 , piece_id=CB27PJDJOIHXV2JLVVOY2JXXDZ7HB6JQZTH6YGYNIVTZQMTP4CJQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.750465619 (887h51m44.621548664s) [noise=true , address=145.79.227.13:4804 , piece_id=Z3H3KFSARJMZDAIWXZ7X52ERD73PHQ4EQTBAF7TH4LX6VHPDDLEQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.750839451 (887h51m44.621922486s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.751162165 (887h51m44.62224521s) [noise=true , address=145.79.227.5:4223 , piece_id=6RMRBZNGRFTUMGRDOJANB5CIBYJRU4G5EHZOFSMDSX6UNIKINJHQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.751499277 (887h51m44.622582312s) [noise=true , address=145.79.227.18:4006 , piece_id=RDSUXZAJXHDSQGNZ4BSOIGZYK2DBAOS3OKCXD7EKUXGLVB7SXFVQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.751801051 (887h51m44.622884086s) [noise=true , address=145.79.227.11:4729 , piece_id=OUQFHEARVMKMD3NOSYNK3SGS65GHO2VVWI33TMDSGNZSVI6U2GGQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.752107515 (887h51m44.62319056s) [noise=true , address=45.91.116.13:4156 , piece_id=ZSQICLD2EZCVFCWHWBBPHLBJQJ5EAIYQG2MBQWACDPAX5RDPKXOQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.752505893 (887h51m44.623588938s) [noise=true , address=145.79.227.19:4079 , piece_id=QMKDT3RBPNHPJMUIAJAFPEAYT4YJ7ECHCSGZWDNSBMSSJR34NUFA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.753696261 (887h51m44.624779306s) [noise=true , address=145.79.227.40:4051 , piece_id=3S7ANT2SY7UNVUGT76H4SM3OIC5CY6JTTE2LR3WVOIYWA6IIWB3A, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:47.390523966 (887h51m47.261607001s) [upload=776950]: Next returned nil segment Jan 11 15:20:47.390543012 (887h51m47.261626057s) [upload=776950]: Waiting for error group managing segments... Jan 11 15:20:47.822599671 (887h51m47.693682706s) [noise=true , address=145.79.227.19:4079 , piece_id=QMKDT3RBPNHPJMUIAJAFPEAYT4YJ7ECHCSGZWDNSBMSSJR34NUFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:47.998445353 (887h51m47.869528388s) [noise=true , address=45.91.116.13:4156 , piece_id=ZSQICLD2EZCVFCWHWBBPHLBJQJ5EAIYQG2MBQWACDPAX5RDPKXOQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:48.230600154 (887h51m48.101683199s) [noise=true , address=77.90.31.13:4627 , piece_id=GT4K2RYPOKWXA62QTYDFUVGQC7MMRWXXKB5ZJ6YQIKGKT3SWZEUA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.569294804 (887h51m49.440377839s) [noise=true , address=145.79.227.13:4804 , piece_id=Z3H3KFSARJMZDAIWXZ7X52ERD73PHQ4EQTBAF7TH4LX6VHPDDLEQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.591086838 (887h51m49.462169873s) [noise=true , address=145.79.227.12:4765 , piece_id=JIZAWBTRKZKBBHLTXFURJOBJXLAHF34MYA6ZCJWDTZKEWTKHSA7Q, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.733493227 (887h51m49.604576272s) [noise=true , address=145.79.227.41:4085 , piece_id=CB27PJDJOIHXV2JLVVOY2JXXDZ7HB6JQZTH6YGYNIVTZQMTP4CJQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.794463392 (887h51m49.665546427s) [noise=true , address=145.79.227.17:4972 , piece_id=WWFXI2QSE25XAV3LIXK26YMZURW3HJ7PJWVCZPU6Q3MJ7ZU4AVHA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.967238577 (887h51m49.838321632s) [noise=true , address=145.79.227.18:4006 , piece_id=RDSUXZAJXHDSQGNZ4BSOIGZYK2DBAOS3OKCXD7EKUXGLVB7SXFVQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.989894235 (887h51m49.86097727s) [noise=true , address=145.79.227.22:4509 , piece_id=SAASLZ536HTQD5SSEK6XCLZLQHBQMU7XS5634662DDLIOK4SP4IQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:50.725655336 (887h51m50.596738371s) [noise=true , address=145.79.227.40:4051 , piece_id=3S7ANT2SY7UNVUGT76H4SM3OIC5CY6JTTE2LR3WVOIYWA6IIWB3A, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:53.061473450 (887h51m52.932556485s) [noise=true , address=145.79.227.5:4223 , piece_id=6RMRBZNGRFTUMGRDOJANB5CIBYJRU4G5EHZOFSMDSX6UNIKINJHQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:53.664200566 (887h51m53.535283601s) [noise=true , address=145.79.227.33:4869 , piece_id=CH2RKQPDTDMB4XC4GT6I22SLZWSGZYCAUBSP3EAGN66KNNKODEFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:54.439611625 (887h51m54.31069466s) [noise=true , address=145.79.227.11:4729 , piece_id=OUQFHEARVMKMD3NOSYNK3SGS65GHO2VVWI33TMDSGNZSVI6U2GGQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:54.439630861 (887h51m54.310713896s) [seg_pos={0 0}, upload=776950]: Segment reached optimal threshold of 13 pieces. Jan 11 15:20:54.439937875 (887h51m54.311020911s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF Jan 11 15:20:54.439980266 (887h51m54.311063311s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold Jan 11 15:20:54.439998441 (887h51m54.311081476s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: read tcp 188.166.225.170:41126->145.79.227.49:4337: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 15:20:54.440006246 (887h51m54.311089281s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold Jan 11 15:20:54.440191488 (887h51m54.311274523s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1): protocol: read tcp 188.166.225.170:36086->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 15:20:54.440199785 (887h51m54.31128282s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold Jan 11 15:20:54.440216286 (887h51m54.311299321s) [seg_pos={0 0}, upload=776950]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 15:20:54.440321306 (887h51m54.311404341s) [upload=776950]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 15:20:54.440326495 (887h51m54.31140953s) [upload=776950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 15:20:54.726699085 (887h51m54.59778212s) [upload=776950]: Done uploading segments... Jan 11 15:20:54.726719413 (887h51m54.597802458s) [upload=776950]: Upload finished. err: <nil> Duration: 5.869251185s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.260349809s trace: https://jaeger.dev.storj.io/trace/7fd32fc43d58b31c FAIL (2026-01-11 15:38:53 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.260377752s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.XX.X..X......X.X.....................X........X......X.X...] duration: 29.974848585s trace: https://jaeger.dev.storj.io/trace/627dde93e011c19f FAIL (2026-01-11 15:38:50 UTC) orbiter warning: context deadline exceeded Duration: 29.974876026s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [XXXXXXX.XXXXXXXXX.X.XX.XX...X..XXX......XXXXX...X.XXXXXXXXXX] duration: 30.000963633s trace: https://jaeger.dev.storj.io/trace/a984e85ad492827 FAIL (2026-01-11 15:38:42 UTC) orbiter warning: context deadline exceeded Duration: 30.000974874s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.458188731s trace: https://jaeger.dev.storj.io/trace/db08c1a98355252 FAIL (2026-01-11 15:38:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, Piece ID: D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA) context deadline exceeded; piecestore: (Node ID: 1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, Piece ID: IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ) context deadline exceeded; piecestore: (Node ID: 12iDqXUpUqsDwhzKGNY1kMqEhS6vbHiESooVR9xu92B3SGybNmn, Piece ID: 3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ) context deadline exceeded ======LOGS====== =ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.515238933 (887h51m9.386321968s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.515935319 (887h51m9.387018354s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.516392410 (887h51m9.387475455s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.516885660 (887h51m9.387968705s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.517268609 (887h51m9.388351654s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.517740817 (887h51m9.388823852s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.518341942 (887h51m9.389424977s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.518779846 (887h51m9.389862881s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.519286630 (887h51m9.390369676s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.521517579 (887h51m9.392600614s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.524918765 (887h51m9.39600181s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.527183728 (887h51m9.398266783s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.529232859 (887h51m9.400315894s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.532984954 (887h51m9.404067999s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.536582354 (887h51m9.407665399s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.539559423 (887h51m9.410642468s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:10.837257700 (887h51m10.708340735s) [upload=776945]: Next returned nil segment Jan 11 15:20:10.837276767 (887h51m10.708359812s) [upload=776945]: Waiting for error group managing segments... Jan 11 15:20:11.321612300 (887h51m11.192695346s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:12.321151387 (887h51m12.192234433s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:12.481216511 (887h51m12.352299546s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:12.632251410 (887h51m12.503334445s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.302406869 (887h51m13.173489904s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.417486975 (887h51m13.28857001s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.437898801 (887h51m13.308981846s) [noise=true , address=145.79.227.33:4897 , piece_id=ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.666780260 (887h51m13.537863305s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.677746121 (887h51m13.548829156s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:14.339206281 (887h51m14.210289316s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:14.557434685 (887h51m14.42851773s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:15.047052989 (887h51m14.918136024s) [noise=true , address=77.90.31.11:4574 , piece_id=D6H557N23KO5L32TEYGYNSY7YHRPTW4VSN56ZLQYYAI5R6RN3FSA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:15.347734503 (887h51m15.218817548s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:15.431090403 (887h51m15.302173448s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:15.431113807 (887h51m15.302196852s) [seg_pos={0 0}, upload=776945]: Segment reached optimal threshold of 14 pieces. Jan 11 15:20:15.431355577 (887h51m15.302438622s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cCPRmsdkrVKFyMfLdBNYhnDhcA2Los7kbvuYUT6dcofPLVXZ7): protocol: expected piece hash; context canceled; EOF Jan 11 15:20:15.431362331 (887h51m15.302445376s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold Jan 11 15:20:15.431472140 (887h51m15.302555175s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Jan 11 15:20:15.431491286 (887h51m15.302574321s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold Jan 11 15:20:15.431505644 (887h51m15.302588679s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9): protocol: expected piece hash; context canceled; EOF Jan 11 15:20:15.431513218 (887h51m15.302596263s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold Jan 11 15:20:15.431640601 (887h51m15.302723646s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: read tcp 188.166.225.170:45042->145.79.227.30:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 15:20:15.431654497 (887h51m15.302737572s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold Jan 11 15:20:15.431749248 (887h51m15.302832283s) [seg_pos={0 0}, upload=776945]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 11 15:20:15.431840110 (887h51m15.302923155s) [upload=776945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 15:20:15.431847053 (887h51m15.302930088s) [upload=776945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 15:20:15.728771354 (887h51m15.599854419s) [upload=776945]: Done uploading segments... Jan 11 15:20:15.728795108 (887h51m15.599878153s) [upload=776945]: Upload finished. err: <nil> Duration: 5.458220682s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.040973986s trace: https://jaeger.dev.storj.io/trace/c31ec8b6bb524c FAIL (2026-01-11 15:38:26 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.040995305s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [XX.X.XXXXXXXXXXXX.XXXXX.XXXXXXXXXX.XX.XXXXX.XXXXX.X.XXX.XXX.] duration: 3.96686796s trace: https://jaeger.dev.storj.io/trace/1f0c71cc5c1bd53a FAIL (2026-01-11 15:38:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.966954725s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [XXXXXXXX.XXXXXXXXX.X.XX.XX...X..XXX......XXXXX...X.XXXXXXXXX] duration: 30.000953598s trace: https://jaeger.dev.storj.io/trace/57cbf0f941888ac4 FAIL (2026-01-11 15:38:12 UTC) orbiter warning: context deadline exceeded Duration: 30.000983715s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................X...X....XX.......................] duration: 5.871576819s trace: https://jaeger.dev.storj.io/trace/15f47b4548974807 FAIL (2026-01-11 15:38:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 11 15:20:44.499817824 (887h51m44.37090086s) [upload=776950]: Starting upload Jan 11 15:20:44.499886175 (887h51m44.37096921s) [upload=776950]: Done starting upload Jan 11 15:20:44.499998148 (887h51m44.371081183s) [upload=776950]: Uploading segments... Jan 11 15:20:44.500806667 (887h51m44.371889712s) [upload=776950]: Got next segment. Inline: false Jan 11 15:20:44.500861151 (887h51m44.371944186s) [upload=776950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 15:20:44.500866932 (887h51m44.371949987s) [upload=776950]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 15:20:44.748047886 (887h51m44.619130931s) [seg_pos={0 0}, upload=776950]: Begin upload segment... Jan 11 15:20:44.748112808 (887h51m44.619195843s) [seg_pos={0 0}, upload=776950]: Done begin upload segment. Jan 11 15:20:44.748136734 (887h51m44.619219779s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.748152184 (887h51m44.619235249s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.748307379 (887h51m44.619390414s) [noise=true , address=145.79.227.33:4869 , piece_id=CH2RKQPDTDMB4XC4GT6I22SLZWSGZYCAUBSP3EAGN66KNNKODEFA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.748774909 (887h51m44.619857944s) [noise=true , address=145.79.227.12:4765 , piece_id=JIZAWBTRKZKBBHLTXFURJOBJXLAHF34MYA6ZCJWDTZKEWTKHSA7Q, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.749107543 (887h51m44.620190578s) [noise=true , address=145.79.227.22:4509 , piece_id=SAASLZ536HTQD5SSEK6XCLZLQHBQMU7XS5634662DDLIOK4SP4IQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.749386073 (887h51m44.620469118s) [noise=true , address=77.90.31.13:4627 , piece_id=GT4K2RYPOKWXA62QTYDFUVGQC7MMRWXXKB5ZJ6YQIKGKT3SWZEUA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.749723326 (887h51m44.620806371s) [noise=true , address=145.79.227.17:4972 , piece_id=WWFXI2QSE25XAV3LIXK26YMZURW3HJ7PJWVCZPU6Q3MJ7ZU4AVHA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.750167912 (887h51m44.621250947s) [noise=true , address=145.79.227.41:4085 , piece_id=CB27PJDJOIHXV2JLVVOY2JXXDZ7HB6JQZTH6YGYNIVTZQMTP4CJQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.750465619 (887h51m44.621548664s) [noise=true , address=145.79.227.13:4804 , piece_id=Z3H3KFSARJMZDAIWXZ7X52ERD73PHQ4EQTBAF7TH4LX6VHPDDLEQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.750839451 (887h51m44.621922486s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.751162165 (887h51m44.62224521s) [noise=true , address=145.79.227.5:4223 , piece_id=6RMRBZNGRFTUMGRDOJANB5CIBYJRU4G5EHZOFSMDSX6UNIKINJHQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.751499277 (887h51m44.622582312s) [noise=true , address=145.79.227.18:4006 , piece_id=RDSUXZAJXHDSQGNZ4BSOIGZYK2DBAOS3OKCXD7EKUXGLVB7SXFVQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.751801051 (887h51m44.622884086s) [noise=true , address=145.79.227.11:4729 , piece_id=OUQFHEARVMKMD3NOSYNK3SGS65GHO2VVWI33TMDSGNZSVI6U2GGQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.752107515 (887h51m44.62319056s) [noise=true , address=45.91.116.13:4156 , piece_id=ZSQICLD2EZCVFCWHWBBPHLBJQJ5EAIYQG2MBQWACDPAX5RDPKXOQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.752505893 (887h51m44.623588938s) [noise=true , address=145.79.227.19:4079 , piece_id=QMKDT3RBPNHPJMUIAJAFPEAYT4YJ7ECHCSGZWDNSBMSSJR34NUFA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.753696261 (887h51m44.624779306s) [noise=true , address=145.79.227.40:4051 , piece_id=3S7ANT2SY7UNVUGT76H4SM3OIC5CY6JTTE2LR3WVOIYWA6IIWB3A, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:47.390523966 (887h51m47.261607001s) [upload=776950]: Next returned nil segment Jan 11 15:20:47.390543012 (887h51m47.261626057s) [upload=776950]: Waiting for error group managing segments... Jan 11 15:20:47.822599671 (887h51m47.693682706s) [noise=true , address=145.79.227.19:4079 , piece_id=QMKDT3RBPNHPJMUIAJAFPEAYT4YJ7ECHCSGZWDNSBMSSJR34NUFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:47.998445353 (887h51m47.869528388s) [noise=true , address=45.91.116.13:4156 , piece_id=ZSQICLD2EZCVFCWHWBBPHLBJQJ5EAIYQG2MBQWACDPAX5RDPKXOQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:48.230600154 (887h51m48.101683199s) [noise=true , address=77.90.31.13:4627 , piece_id=GT4K2RYPOKWXA62QTYDFUVGQC7MMRWXXKB5ZJ6YQIKGKT3SWZEUA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.569294804 (887h51m49.440377839s) [noise=true , address=145.79.227.13:4804 , piece_id=Z3H3KFSARJMZDAIWXZ7X52ERD73PHQ4EQTBAF7TH4LX6VHPDDLEQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.591086838 (887h51m49.462169873s) [noise=true , address=145.79.227.12:4765 , piece_id=JIZAWBTRKZKBBHLTXFURJOBJXLAHF34MYA6ZCJWDTZKEWTKHSA7Q, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.733493227 (887h51m49.604576272s) [noise=true , address=145.79.227.41:4085 , piece_id=CB27PJDJOIHXV2JLVVOY2JXXDZ7HB6JQZTH6YGYNIVTZQMTP4CJQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.794463392 (887h51m49.665546427s) [noise=true , address=145.79.227.17:4972 , piece_id=WWFXI2QSE25XAV3LIXK26YMZURW3HJ7PJWVCZPU6Q3MJ7ZU4AVHA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.967238577 (887h51m49.838321632s) [noise=true , address=145.79.227.18:4006 , piece_id=RDSUXZAJXHDSQGNZ4BSOIGZYK2DBAOS3OKCXD7EKUXGLVB7SXFVQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.989894235 (887h51m49.86097727s) [noise=true , address=145.79.227.22:4509 , piece_id=SAASLZ536HTQD5SSEK6XCLZLQHBQMU7XS5634662DDLIOK4SP4IQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:50.725655336 (887h51m50.596738371s) [noise=true , address=145.79.227.40:4051 , piece_id=3S7ANT2SY7UNVUGT76H4SM3OIC5CY6JTTE2LR3WVOIYWA6IIWB3A, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:53.061473450 (887h51m52.932556485s) [noise=true , address=145.79.227.5:4223 , piece_id=6RMRBZNGRFTUMGRDOJANB5CIBYJRU4G5EHZOFSMDSX6UNIKINJHQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:53.664200566 (887h51m53.535283601s) [noise=true , address=145.79.227.33:4869 , piece_id=CH2RKQPDTDMB4XC4GT6I22SLZWSGZYCAUBSP3EAGN66KNNKODEFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:54.439611625 (887h51m54.31069466s) [noise=true , address=145.79.227.11:4729 , piece_id=OUQFHEARVMKMD3NOSYNK3SGS65GHO2VVWI33TMDSGNZSVI6U2GGQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:54.439630861 (887h51m54.310713896s) [seg_pos={0 0}, upload=776950]: Segment reached optimal threshold of 13 pieces. Jan 11 15:20:54.439937875 (887h51m54.311020911s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF Jan 11 15:20:54.439980266 (887h51m54.311063311s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold Jan 11 15:20:54.439998441 (887h51m54.311081476s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: read tcp 188.166.225.170:41126->145.79.227.49:4337: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 15:20:54.440006246 (887h51m54.311089281s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold Jan 11 15:20:54.440191488 (887h51m54.311274523s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1): protocol: read tcp 188.166.225.170:36086->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 15:20:54.440199785 (887h51m54.31128282s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold Jan 11 15:20:54.440216286 (887h51m54.311299321s) [seg_pos={0 0}, upload=776950]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 15:20:54.440321306 (887h51m54.311404341s) [upload=776950]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 15:20:54.440326495 (887h51m54.31140953s) [upload=776950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 15:20:54.726699085 (887h51m54.59778212s) [upload=776950]: Done uploading segments... Jan 11 15:20:54.726719413 (887h51m54.597802458s) [upload=776950]: Upload finished. err: <nil> Duration: 5.871604533s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.280351261s trace: https://jaeger.dev.storj.io/trace/7fd32fc43d58b31c FAIL (2026-01-11 15:38:02 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.280404662s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [XXX.X.XXXXXXXXXXXX.XXXXX.XXXXXXXXXX.XX.XXXXX.XXXXX.X.XXX.XXX] duration: 5.000971838s trace: https://jaeger.dev.storj.io/trace/1f0c71cc5c1bd53a FAIL (2026-01-11 15:38:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000997436s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [..XX.X..X......X.X.....................X........X......X.X..] duration: 26.644553053s trace: https://jaeger.dev.storj.io/trace/627dde93e011c19f FAIL (2026-01-11 15:37:58 UTC) orbiter warning: context deadline exceeded Duration: 26.64457789s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................X...X....XX......................] duration: 5.838301639s trace: https://jaeger.dev.storj.io/trace/15f47b4548974807 FAIL (2026-01-11 15:37:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 11 15:20:44.499817824 (887h51m44.37090086s) [upload=776950]: Starting upload Jan 11 15:20:44.499886175 (887h51m44.37096921s) [upload=776950]: Done starting upload Jan 11 15:20:44.499998148 (887h51m44.371081183s) [upload=776950]: Uploading segments... Jan 11 15:20:44.500806667 (887h51m44.371889712s) [upload=776950]: Got next segment. Inline: false Jan 11 15:20:44.500861151 (887h51m44.371944186s) [upload=776950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 15:20:44.500866932 (887h51m44.371949987s) [upload=776950]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 15:20:44.748047886 (887h51m44.619130931s) [seg_pos={0 0}, upload=776950]: Begin upload segment... Jan 11 15:20:44.748112808 (887h51m44.619195843s) [seg_pos={0 0}, upload=776950]: Done begin upload segment. Jan 11 15:20:44.748136734 (887h51m44.619219779s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.748152184 (887h51m44.619235249s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.748307379 (887h51m44.619390414s) [noise=true , address=145.79.227.33:4869 , piece_id=CH2RKQPDTDMB4XC4GT6I22SLZWSGZYCAUBSP3EAGN66KNNKODEFA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.748774909 (887h51m44.619857944s) [noise=true , address=145.79.227.12:4765 , piece_id=JIZAWBTRKZKBBHLTXFURJOBJXLAHF34MYA6ZCJWDTZKEWTKHSA7Q, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.749107543 (887h51m44.620190578s) [noise=true , address=145.79.227.22:4509 , piece_id=SAASLZ536HTQD5SSEK6XCLZLQHBQMU7XS5634662DDLIOK4SP4IQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.749386073 (887h51m44.620469118s) [noise=true , address=77.90.31.13:4627 , piece_id=GT4K2RYPOKWXA62QTYDFUVGQC7MMRWXXKB5ZJ6YQIKGKT3SWZEUA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.749723326 (887h51m44.620806371s) [noise=true , address=145.79.227.17:4972 , piece_id=WWFXI2QSE25XAV3LIXK26YMZURW3HJ7PJWVCZPU6Q3MJ7ZU4AVHA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.750167912 (887h51m44.621250947s) [noise=true , address=145.79.227.41:4085 , piece_id=CB27PJDJOIHXV2JLVVOY2JXXDZ7HB6JQZTH6YGYNIVTZQMTP4CJQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.750465619 (887h51m44.621548664s) [noise=true , address=145.79.227.13:4804 , piece_id=Z3H3KFSARJMZDAIWXZ7X52ERD73PHQ4EQTBAF7TH4LX6VHPDDLEQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.750839451 (887h51m44.621922486s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.751162165 (887h51m44.62224521s) [noise=true , address=145.79.227.5:4223 , piece_id=6RMRBZNGRFTUMGRDOJANB5CIBYJRU4G5EHZOFSMDSX6UNIKINJHQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.751499277 (887h51m44.622582312s) [noise=true , address=145.79.227.18:4006 , piece_id=RDSUXZAJXHDSQGNZ4BSOIGZYK2DBAOS3OKCXD7EKUXGLVB7SXFVQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.751801051 (887h51m44.622884086s) [noise=true , address=145.79.227.11:4729 , piece_id=OUQFHEARVMKMD3NOSYNK3SGS65GHO2VVWI33TMDSGNZSVI6U2GGQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.752107515 (887h51m44.62319056s) [noise=true , address=45.91.116.13:4156 , piece_id=ZSQICLD2EZCVFCWHWBBPHLBJQJ5EAIYQG2MBQWACDPAX5RDPKXOQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.752505893 (887h51m44.623588938s) [noise=true , address=145.79.227.19:4079 , piece_id=QMKDT3RBPNHPJMUIAJAFPEAYT4YJ7ECHCSGZWDNSBMSSJR34NUFA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.753696261 (887h51m44.624779306s) [noise=true , address=145.79.227.40:4051 , piece_id=3S7ANT2SY7UNVUGT76H4SM3OIC5CY6JTTE2LR3WVOIYWA6IIWB3A, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:47.390523966 (887h51m47.261607001s) [upload=776950]: Next returned nil segment Jan 11 15:20:47.390543012 (887h51m47.261626057s) [upload=776950]: Waiting for error group managing segments... Jan 11 15:20:47.822599671 (887h51m47.693682706s) [noise=true , address=145.79.227.19:4079 , piece_id=QMKDT3RBPNHPJMUIAJAFPEAYT4YJ7ECHCSGZWDNSBMSSJR34NUFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:47.998445353 (887h51m47.869528388s) [noise=true , address=45.91.116.13:4156 , piece_id=ZSQICLD2EZCVFCWHWBBPHLBJQJ5EAIYQG2MBQWACDPAX5RDPKXOQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:48.230600154 (887h51m48.101683199s) [noise=true , address=77.90.31.13:4627 , piece_id=GT4K2RYPOKWXA62QTYDFUVGQC7MMRWXXKB5ZJ6YQIKGKT3SWZEUA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.569294804 (887h51m49.440377839s) [noise=true , address=145.79.227.13:4804 , piece_id=Z3H3KFSARJMZDAIWXZ7X52ERD73PHQ4EQTBAF7TH4LX6VHPDDLEQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.591086838 (887h51m49.462169873s) [noise=true , address=145.79.227.12:4765 , piece_id=JIZAWBTRKZKBBHLTXFURJOBJXLAHF34MYA6ZCJWDTZKEWTKHSA7Q, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.733493227 (887h51m49.604576272s) [noise=true , address=145.79.227.41:4085 , piece_id=CB27PJDJOIHXV2JLVVOY2JXXDZ7HB6JQZTH6YGYNIVTZQMTP4CJQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.794463392 (887h51m49.665546427s) [noise=true , address=145.79.227.17:4972 , piece_id=WWFXI2QSE25XAV3LIXK26YMZURW3HJ7PJWVCZPU6Q3MJ7ZU4AVHA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.967238577 (887h51m49.838321632s) [noise=true , address=145.79.227.18:4006 , piece_id=RDSUXZAJXHDSQGNZ4BSOIGZYK2DBAOS3OKCXD7EKUXGLVB7SXFVQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.989894235 (887h51m49.86097727s) [noise=true , address=145.79.227.22:4509 , piece_id=SAASLZ536HTQD5SSEK6XCLZLQHBQMU7XS5634662DDLIOK4SP4IQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:50.725655336 (887h51m50.596738371s) [noise=true , address=145.79.227.40:4051 , piece_id=3S7ANT2SY7UNVUGT76H4SM3OIC5CY6JTTE2LR3WVOIYWA6IIWB3A, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:53.061473450 (887h51m52.932556485s) [noise=true , address=145.79.227.5:4223 , piece_id=6RMRBZNGRFTUMGRDOJANB5CIBYJRU4G5EHZOFSMDSX6UNIKINJHQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:53.664200566 (887h51m53.535283601s) [noise=true , address=145.79.227.33:4869 , piece_id=CH2RKQPDTDMB4XC4GT6I22SLZWSGZYCAUBSP3EAGN66KNNKODEFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:54.439611625 (887h51m54.31069466s) [noise=true , address=145.79.227.11:4729 , piece_id=OUQFHEARVMKMD3NOSYNK3SGS65GHO2VVWI33TMDSGNZSVI6U2GGQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:54.439630861 (887h51m54.310713896s) [seg_pos={0 0}, upload=776950]: Segment reached optimal threshold of 13 pieces. Jan 11 15:20:54.439937875 (887h51m54.311020911s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF Jan 11 15:20:54.439980266 (887h51m54.311063311s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold Jan 11 15:20:54.439998441 (887h51m54.311081476s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: read tcp 188.166.225.170:41126->145.79.227.49:4337: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 15:20:54.440006246 (887h51m54.311089281s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold Jan 11 15:20:54.440191488 (887h51m54.311274523s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1): protocol: read tcp 188.166.225.170:36086->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 15:20:54.440199785 (887h51m54.31128282s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold Jan 11 15:20:54.440216286 (887h51m54.311299321s) [seg_pos={0 0}, upload=776950]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 15:20:54.440321306 (887h51m54.311404341s) [upload=776950]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 15:20:54.440326495 (887h51m54.31140953s) [upload=776950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 15:20:54.726699085 (887h51m54.59778212s) [upload=776950]: Done uploading segments... Jan 11 15:20:54.726719413 (887h51m54.597802458s) [upload=776950]: Upload finished. err: <nil> Duration: 5.838328849s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [XXXX.X.XXXXXXXXXXXX.XXXXX.XXXXXXXXXX.XX.XXXXX.XXXXX.X.XXX.XX] duration: 5.000595182s trace: https://jaeger.dev.storj.io/trace/155a00fff18b2203 FAIL (2026-01-11 15:37:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000613606s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.040819101s trace: https://jaeger.dev.storj.io/trace/c31ec8b6bb524c FAIL (2026-01-11 15:37:27 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.040835153s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.436114584s trace: https://jaeger.dev.storj.io/trace/7fd32fc43d58b31c FAIL (2026-01-11 15:37:26 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.436167704s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.852857915s trace: https://jaeger.dev.storj.io/trace/db08c1a98355252 FAIL (2026-01-11 15:37:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, Piece ID: D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA) context deadline exceeded; piecestore: (Node ID: 1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, Piece ID: IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ) context deadline exceeded; piecestore: (Node ID: 12iDqXUpUqsDwhzKGNY1kMqEhS6vbHiESooVR9xu92B3SGybNmn, Piece ID: 3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ) context deadline exceeded ======LOGS====== =ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.515238933 (887h51m9.386321968s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.515935319 (887h51m9.387018354s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.516392410 (887h51m9.387475455s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.516885660 (887h51m9.387968705s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.517268609 (887h51m9.388351654s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.517740817 (887h51m9.388823852s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.518341942 (887h51m9.389424977s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.518779846 (887h51m9.389862881s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.519286630 (887h51m9.390369676s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.521517579 (887h51m9.392600614s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.524918765 (887h51m9.39600181s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.527183728 (887h51m9.398266783s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.529232859 (887h51m9.400315894s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.532984954 (887h51m9.404067999s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.536582354 (887h51m9.407665399s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.539559423 (887h51m9.410642468s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:10.837257700 (887h51m10.708340735s) [upload=776945]: Next returned nil segment Jan 11 15:20:10.837276767 (887h51m10.708359812s) [upload=776945]: Waiting for error group managing segments... Jan 11 15:20:11.321612300 (887h51m11.192695346s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:12.321151387 (887h51m12.192234433s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:12.481216511 (887h51m12.352299546s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:12.632251410 (887h51m12.503334445s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.302406869 (887h51m13.173489904s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.417486975 (887h51m13.28857001s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.437898801 (887h51m13.308981846s) [noise=true , address=145.79.227.33:4897 , piece_id=ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.666780260 (887h51m13.537863305s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.677746121 (887h51m13.548829156s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:14.339206281 (887h51m14.210289316s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:14.557434685 (887h51m14.42851773s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:15.047052989 (887h51m14.918136024s) [noise=true , address=77.90.31.11:4574 , piece_id=D6H557N23KO5L32TEYGYNSY7YHRPTW4VSN56ZLQYYAI5R6RN3FSA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:15.347734503 (887h51m15.218817548s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:15.431090403 (887h51m15.302173448s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:15.431113807 (887h51m15.302196852s) [seg_pos={0 0}, upload=776945]: Segment reached optimal threshold of 14 pieces. Jan 11 15:20:15.431355577 (887h51m15.302438622s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cCPRmsdkrVKFyMfLdBNYhnDhcA2Los7kbvuYUT6dcofPLVXZ7): protocol: expected piece hash; context canceled; EOF Jan 11 15:20:15.431362331 (887h51m15.302445376s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold Jan 11 15:20:15.431472140 (887h51m15.302555175s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Jan 11 15:20:15.431491286 (887h51m15.302574321s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold Jan 11 15:20:15.431505644 (887h51m15.302588679s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9): protocol: expected piece hash; context canceled; EOF Jan 11 15:20:15.431513218 (887h51m15.302596263s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold Jan 11 15:20:15.431640601 (887h51m15.302723646s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: read tcp 188.166.225.170:45042->145.79.227.30:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 15:20:15.431654497 (887h51m15.302737572s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold Jan 11 15:20:15.431749248 (887h51m15.302832283s) [seg_pos={0 0}, upload=776945]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 11 15:20:15.431840110 (887h51m15.302923155s) [upload=776945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 15:20:15.431847053 (887h51m15.302930088s) [upload=776945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 15:20:15.728771354 (887h51m15.599854419s) [upload=776945]: Done uploading segments... Jan 11 15:20:15.728795108 (887h51m15.599878153s) [upload=776945]: Upload finished. err: <nil> Duration: 5.852884014s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [XXXXXXXXX.XXXXXXXXX.X.XX.XX...X..XXX......XXXXX...X.XXXXXXXX] duration: 30.001591669s trace: https://jaeger.dev.storj.io/trace/1e685e50ab8c2269 FAIL (2026-01-11 15:37:11 UTC) orbiter warning: context deadline exceeded Duration: 30.001634039s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [...XX.X..X......X.X.....................X........X......X.X.] duration: 48.214512623s trace: https://jaeger.dev.storj.io/trace/627dde93e011c19f FAIL (2026-01-11 15:37:09 UTC) orbiter warning: context deadline exceeded Duration: 48.214530107s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.705153616s trace: https://jaeger.dev.storj.io/trace/db08c1a98355252 FAIL (2026-01-11 15:37:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, Piece ID: D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA) context deadline exceeded; piecestore: (Node ID: 1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, Piece ID: IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ) context deadline exceeded; piecestore: (Node ID: 12iDqXUpUqsDwhzKGNY1kMqEhS6vbHiESooVR9xu92B3SGybNmn, Piece ID: 3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ) context deadline exceeded ======LOGS====== =ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.515238933 (887h51m9.386321968s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.515935319 (887h51m9.387018354s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.516392410 (887h51m9.387475455s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.516885660 (887h51m9.387968705s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.517268609 (887h51m9.388351654s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.517740817 (887h51m9.388823852s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.518341942 (887h51m9.389424977s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.518779846 (887h51m9.389862881s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.519286630 (887h51m9.390369676s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.521517579 (887h51m9.392600614s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.524918765 (887h51m9.39600181s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.527183728 (887h51m9.398266783s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.529232859 (887h51m9.400315894s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.532984954 (887h51m9.404067999s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.536582354 (887h51m9.407665399s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.539559423 (887h51m9.410642468s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:10.837257700 (887h51m10.708340735s) [upload=776945]: Next returned nil segment Jan 11 15:20:10.837276767 (887h51m10.708359812s) [upload=776945]: Waiting for error group managing segments... Jan 11 15:20:11.321612300 (887h51m11.192695346s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:12.321151387 (887h51m12.192234433s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:12.481216511 (887h51m12.352299546s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:12.632251410 (887h51m12.503334445s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.302406869 (887h51m13.173489904s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.417486975 (887h51m13.28857001s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.437898801 (887h51m13.308981846s) [noise=true , address=145.79.227.33:4897 , piece_id=ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.666780260 (887h51m13.537863305s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.677746121 (887h51m13.548829156s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:14.339206281 (887h51m14.210289316s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:14.557434685 (887h51m14.42851773s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:15.047052989 (887h51m14.918136024s) [noise=true , address=77.90.31.11:4574 , piece_id=D6H557N23KO5L32TEYGYNSY7YHRPTW4VSN56ZLQYYAI5R6RN3FSA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:15.347734503 (887h51m15.218817548s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:15.431090403 (887h51m15.302173448s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:15.431113807 (887h51m15.302196852s) [seg_pos={0 0}, upload=776945]: Segment reached optimal threshold of 14 pieces. Jan 11 15:20:15.431355577 (887h51m15.302438622s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cCPRmsdkrVKFyMfLdBNYhnDhcA2Los7kbvuYUT6dcofPLVXZ7): protocol: expected piece hash; context canceled; EOF Jan 11 15:20:15.431362331 (887h51m15.302445376s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold Jan 11 15:20:15.431472140 (887h51m15.302555175s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Jan 11 15:20:15.431491286 (887h51m15.302574321s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold Jan 11 15:20:15.431505644 (887h51m15.302588679s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9): protocol: expected piece hash; context canceled; EOF Jan 11 15:20:15.431513218 (887h51m15.302596263s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold Jan 11 15:20:15.431640601 (887h51m15.302723646s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: read tcp 188.166.225.170:45042->145.79.227.30:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 15:20:15.431654497 (887h51m15.302737572s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold Jan 11 15:20:15.431749248 (887h51m15.302832283s) [seg_pos={0 0}, upload=776945]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 11 15:20:15.431840110 (887h51m15.302923155s) [upload=776945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 15:20:15.431847053 (887h51m15.302930088s) [upload=776945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 15:20:15.728771354 (887h51m15.599854419s) [upload=776945]: Done uploading segments... Jan 11 15:20:15.728795108 (887h51m15.599878153s) [upload=776945]: Upload finished. err: <nil> Duration: 5.705182421s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................X...X....XX.....................] duration: 5.712024199s trace: https://jaeger.dev.storj.io/trace/15f47b4548974807 FAIL (2026-01-11 15:36:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 11 15:20:44.499817824 (887h51m44.37090086s) [upload=776950]: Starting upload Jan 11 15:20:44.499886175 (887h51m44.37096921s) [upload=776950]: Done starting upload Jan 11 15:20:44.499998148 (887h51m44.371081183s) [upload=776950]: Uploading segments... Jan 11 15:20:44.500806667 (887h51m44.371889712s) [upload=776950]: Got next segment. Inline: false Jan 11 15:20:44.500861151 (887h51m44.371944186s) [upload=776950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 15:20:44.500866932 (887h51m44.371949987s) [upload=776950]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 15:20:44.748047886 (887h51m44.619130931s) [seg_pos={0 0}, upload=776950]: Begin upload segment... Jan 11 15:20:44.748112808 (887h51m44.619195843s) [seg_pos={0 0}, upload=776950]: Done begin upload segment. Jan 11 15:20:44.748136734 (887h51m44.619219779s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.748152184 (887h51m44.619235249s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.748307379 (887h51m44.619390414s) [noise=true , address=145.79.227.33:4869 , piece_id=CH2RKQPDTDMB4XC4GT6I22SLZWSGZYCAUBSP3EAGN66KNNKODEFA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.748774909 (887h51m44.619857944s) [noise=true , address=145.79.227.12:4765 , piece_id=JIZAWBTRKZKBBHLTXFURJOBJXLAHF34MYA6ZCJWDTZKEWTKHSA7Q, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.749107543 (887h51m44.620190578s) [noise=true , address=145.79.227.22:4509 , piece_id=SAASLZ536HTQD5SSEK6XCLZLQHBQMU7XS5634662DDLIOK4SP4IQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.749386073 (887h51m44.620469118s) [noise=true , address=77.90.31.13:4627 , piece_id=GT4K2RYPOKWXA62QTYDFUVGQC7MMRWXXKB5ZJ6YQIKGKT3SWZEUA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.749723326 (887h51m44.620806371s) [noise=true , address=145.79.227.17:4972 , piece_id=WWFXI2QSE25XAV3LIXK26YMZURW3HJ7PJWVCZPU6Q3MJ7ZU4AVHA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.750167912 (887h51m44.621250947s) [noise=true , address=145.79.227.41:4085 , piece_id=CB27PJDJOIHXV2JLVVOY2JXXDZ7HB6JQZTH6YGYNIVTZQMTP4CJQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.750465619 (887h51m44.621548664s) [noise=true , address=145.79.227.13:4804 , piece_id=Z3H3KFSARJMZDAIWXZ7X52ERD73PHQ4EQTBAF7TH4LX6VHPDDLEQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.750839451 (887h51m44.621922486s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.751162165 (887h51m44.62224521s) [noise=true , address=145.79.227.5:4223 , piece_id=6RMRBZNGRFTUMGRDOJANB5CIBYJRU4G5EHZOFSMDSX6UNIKINJHQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.751499277 (887h51m44.622582312s) [noise=true , address=145.79.227.18:4006 , piece_id=RDSUXZAJXHDSQGNZ4BSOIGZYK2DBAOS3OKCXD7EKUXGLVB7SXFVQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.751801051 (887h51m44.622884086s) [noise=true , address=145.79.227.11:4729 , piece_id=OUQFHEARVMKMD3NOSYNK3SGS65GHO2VVWI33TMDSGNZSVI6U2GGQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.752107515 (887h51m44.62319056s) [noise=true , address=45.91.116.13:4156 , piece_id=ZSQICLD2EZCVFCWHWBBPHLBJQJ5EAIYQG2MBQWACDPAX5RDPKXOQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.752505893 (887h51m44.623588938s) [noise=true , address=145.79.227.19:4079 , piece_id=QMKDT3RBPNHPJMUIAJAFPEAYT4YJ7ECHCSGZWDNSBMSSJR34NUFA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.753696261 (887h51m44.624779306s) [noise=true , address=145.79.227.40:4051 , piece_id=3S7ANT2SY7UNVUGT76H4SM3OIC5CY6JTTE2LR3WVOIYWA6IIWB3A, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:47.390523966 (887h51m47.261607001s) [upload=776950]: Next returned nil segment Jan 11 15:20:47.390543012 (887h51m47.261626057s) [upload=776950]: Waiting for error group managing segments... Jan 11 15:20:47.822599671 (887h51m47.693682706s) [noise=true , address=145.79.227.19:4079 , piece_id=QMKDT3RBPNHPJMUIAJAFPEAYT4YJ7ECHCSGZWDNSBMSSJR34NUFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:47.998445353 (887h51m47.869528388s) [noise=true , address=45.91.116.13:4156 , piece_id=ZSQICLD2EZCVFCWHWBBPHLBJQJ5EAIYQG2MBQWACDPAX5RDPKXOQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:48.230600154 (887h51m48.101683199s) [noise=true , address=77.90.31.13:4627 , piece_id=GT4K2RYPOKWXA62QTYDFUVGQC7MMRWXXKB5ZJ6YQIKGKT3SWZEUA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.569294804 (887h51m49.440377839s) [noise=true , address=145.79.227.13:4804 , piece_id=Z3H3KFSARJMZDAIWXZ7X52ERD73PHQ4EQTBAF7TH4LX6VHPDDLEQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.591086838 (887h51m49.462169873s) [noise=true , address=145.79.227.12:4765 , piece_id=JIZAWBTRKZKBBHLTXFURJOBJXLAHF34MYA6ZCJWDTZKEWTKHSA7Q, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.733493227 (887h51m49.604576272s) [noise=true , address=145.79.227.41:4085 , piece_id=CB27PJDJOIHXV2JLVVOY2JXXDZ7HB6JQZTH6YGYNIVTZQMTP4CJQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.794463392 (887h51m49.665546427s) [noise=true , address=145.79.227.17:4972 , piece_id=WWFXI2QSE25XAV3LIXK26YMZURW3HJ7PJWVCZPU6Q3MJ7ZU4AVHA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.967238577 (887h51m49.838321632s) [noise=true , address=145.79.227.18:4006 , piece_id=RDSUXZAJXHDSQGNZ4BSOIGZYK2DBAOS3OKCXD7EKUXGLVB7SXFVQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.989894235 (887h51m49.86097727s) [noise=true , address=145.79.227.22:4509 , piece_id=SAASLZ536HTQD5SSEK6XCLZLQHBQMU7XS5634662DDLIOK4SP4IQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:50.725655336 (887h51m50.596738371s) [noise=true , address=145.79.227.40:4051 , piece_id=3S7ANT2SY7UNVUGT76H4SM3OIC5CY6JTTE2LR3WVOIYWA6IIWB3A, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:53.061473450 (887h51m52.932556485s) [noise=true , address=145.79.227.5:4223 , piece_id=6RMRBZNGRFTUMGRDOJANB5CIBYJRU4G5EHZOFSMDSX6UNIKINJHQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:53.664200566 (887h51m53.535283601s) [noise=true , address=145.79.227.33:4869 , piece_id=CH2RKQPDTDMB4XC4GT6I22SLZWSGZYCAUBSP3EAGN66KNNKODEFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:54.439611625 (887h51m54.31069466s) [noise=true , address=145.79.227.11:4729 , piece_id=OUQFHEARVMKMD3NOSYNK3SGS65GHO2VVWI33TMDSGNZSVI6U2GGQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:54.439630861 (887h51m54.310713896s) [seg_pos={0 0}, upload=776950]: Segment reached optimal threshold of 13 pieces. Jan 11 15:20:54.439937875 (887h51m54.311020911s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF Jan 11 15:20:54.439980266 (887h51m54.311063311s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold Jan 11 15:20:54.439998441 (887h51m54.311081476s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: read tcp 188.166.225.170:41126->145.79.227.49:4337: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 15:20:54.440006246 (887h51m54.311089281s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold Jan 11 15:20:54.440191488 (887h51m54.311274523s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1): protocol: read tcp 188.166.225.170:36086->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 15:20:54.440199785 (887h51m54.31128282s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold Jan 11 15:20:54.440216286 (887h51m54.311299321s) [seg_pos={0 0}, upload=776950]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 15:20:54.440321306 (887h51m54.311404341s) [upload=776950]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 15:20:54.440326495 (887h51m54.31140953s) [upload=776950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 15:20:54.726699085 (887h51m54.59778212s) [upload=776950]: Done uploading segments... Jan 11 15:20:54.726719413 (887h51m54.597802458s) [upload=776950]: Upload finished. err: <nil> Duration: 5.712052984s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [XXXXX.X.XXXXXXXXXXXX.XXXXX.XXXXXXXXXX.XX.XXXXX.XXXXX.X.XXX.X] duration: 5.001490044s trace: https://jaeger.dev.storj.io/trace/66e03059ad176d91 FAIL (2026-01-11 15:36:54 UTC) orbiter warning: context deadline exceeded Duration: 5.001523407s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [XXXXXXXXXX.XXXXXXXXX.X.XX.XX...X..XXX......XXXXX...X.XXXXXXX] duration: 30.001247032s trace: https://jaeger.dev.storj.io/trace/94a9d551c7d95b9 FAIL (2026-01-11 15:36:41 UTC) orbiter warning: context deadline exceeded Duration: 30.001283061s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 1.953215763s trace: https://jaeger.dev.storj.io/trace/c31ec8b6bb524c FAIL (2026-01-11 15:36:36 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.953244248s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [XXXXXX.X.XXXXXXXXXXXX.XXXXX.XXXXXXXXXX.XX.XXXXX.XXXXX.X.XXX.] duration: 3.882303473s trace: https://jaeger.dev.storj.io/trace/2aba73bfa7cea6cb FAIL (2026-01-11 15:36:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.882325113s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [....XX.X..X......X.X.....................X........X......X.X] duration: 1m0.001643984s trace: https://jaeger.dev.storj.io/trace/627dde93e011c19f FAIL (2026-01-11 15:36:21 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001655335s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.352762582s trace: https://jaeger.dev.storj.io/trace/7fd32fc43d58b31c FAIL (2026-01-11 15:36:13 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.352793439s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.569964648s trace: https://jaeger.dev.storj.io/trace/db08c1a98355252 FAIL (2026-01-11 15:36:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, Piece ID: D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA) context deadline exceeded; piecestore: (Node ID: 1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, Piece ID: IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ) context deadline exceeded; piecestore: (Node ID: 12iDqXUpUqsDwhzKGNY1kMqEhS6vbHiESooVR9xu92B3SGybNmn, Piece ID: 3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ) context deadline exceeded ======LOGS====== =ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.515238933 (887h51m9.386321968s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.515935319 (887h51m9.387018354s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.516392410 (887h51m9.387475455s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.516885660 (887h51m9.387968705s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.517268609 (887h51m9.388351654s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.517740817 (887h51m9.388823852s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.518341942 (887h51m9.389424977s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.518779846 (887h51m9.389862881s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.519286630 (887h51m9.390369676s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.521517579 (887h51m9.392600614s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.524918765 (887h51m9.39600181s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.527183728 (887h51m9.398266783s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.529232859 (887h51m9.400315894s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.532984954 (887h51m9.404067999s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.536582354 (887h51m9.407665399s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.539559423 (887h51m9.410642468s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:10.837257700 (887h51m10.708340735s) [upload=776945]: Next returned nil segment Jan 11 15:20:10.837276767 (887h51m10.708359812s) [upload=776945]: Waiting for error group managing segments... Jan 11 15:20:11.321612300 (887h51m11.192695346s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:12.321151387 (887h51m12.192234433s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:12.481216511 (887h51m12.352299546s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:12.632251410 (887h51m12.503334445s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.302406869 (887h51m13.173489904s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.417486975 (887h51m13.28857001s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.437898801 (887h51m13.308981846s) [noise=true , address=145.79.227.33:4897 , piece_id=ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.666780260 (887h51m13.537863305s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.677746121 (887h51m13.548829156s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:14.339206281 (887h51m14.210289316s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:14.557434685 (887h51m14.42851773s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:15.047052989 (887h51m14.918136024s) [noise=true , address=77.90.31.11:4574 , piece_id=D6H557N23KO5L32TEYGYNSY7YHRPTW4VSN56ZLQYYAI5R6RN3FSA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:15.347734503 (887h51m15.218817548s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:15.431090403 (887h51m15.302173448s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:15.431113807 (887h51m15.302196852s) [seg_pos={0 0}, upload=776945]: Segment reached optimal threshold of 14 pieces. Jan 11 15:20:15.431355577 (887h51m15.302438622s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cCPRmsdkrVKFyMfLdBNYhnDhcA2Los7kbvuYUT6dcofPLVXZ7): protocol: expected piece hash; context canceled; EOF Jan 11 15:20:15.431362331 (887h51m15.302445376s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold Jan 11 15:20:15.431472140 (887h51m15.302555175s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Jan 11 15:20:15.431491286 (887h51m15.302574321s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold Jan 11 15:20:15.431505644 (887h51m15.302588679s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9): protocol: expected piece hash; context canceled; EOF Jan 11 15:20:15.431513218 (887h51m15.302596263s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold Jan 11 15:20:15.431640601 (887h51m15.302723646s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: read tcp 188.166.225.170:45042->145.79.227.30:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 15:20:15.431654497 (887h51m15.302737572s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold Jan 11 15:20:15.431749248 (887h51m15.302832283s) [seg_pos={0 0}, upload=776945]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 11 15:20:15.431840110 (887h51m15.302923155s) [upload=776945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 15:20:15.431847053 (887h51m15.302930088s) [upload=776945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 15:20:15.728771354 (887h51m15.599854419s) [upload=776945]: Done uploading segments... Jan 11 15:20:15.728795108 (887h51m15.599878153s) [upload=776945]: Upload finished. err: <nil> Duration: 5.569992381s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................X...X....XX....................] duration: 5.87969019s trace: https://jaeger.dev.storj.io/trace/15f47b4548974807 FAIL (2026-01-11 15:35:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 11 15:20:44.499817824 (887h51m44.37090086s) [upload=776950]: Starting upload Jan 11 15:20:44.499886175 (887h51m44.37096921s) [upload=776950]: Done starting upload Jan 11 15:20:44.499998148 (887h51m44.371081183s) [upload=776950]: Uploading segments... Jan 11 15:20:44.500806667 (887h51m44.371889712s) [upload=776950]: Got next segment. Inline: false Jan 11 15:20:44.500861151 (887h51m44.371944186s) [upload=776950]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 15:20:44.500866932 (887h51m44.371949987s) [upload=776950]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 15:20:44.748047886 (887h51m44.619130931s) [seg_pos={0 0}, upload=776950]: Begin upload segment... Jan 11 15:20:44.748112808 (887h51m44.619195843s) [seg_pos={0 0}, upload=776950]: Done begin upload segment. Jan 11 15:20:44.748136734 (887h51m44.619219779s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.748152184 (887h51m44.619235249s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.748307379 (887h51m44.619390414s) [noise=true , address=145.79.227.33:4869 , piece_id=CH2RKQPDTDMB4XC4GT6I22SLZWSGZYCAUBSP3EAGN66KNNKODEFA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.748774909 (887h51m44.619857944s) [noise=true , address=145.79.227.12:4765 , piece_id=JIZAWBTRKZKBBHLTXFURJOBJXLAHF34MYA6ZCJWDTZKEWTKHSA7Q, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.749107543 (887h51m44.620190578s) [noise=true , address=145.79.227.22:4509 , piece_id=SAASLZ536HTQD5SSEK6XCLZLQHBQMU7XS5634662DDLIOK4SP4IQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.749386073 (887h51m44.620469118s) [noise=true , address=77.90.31.13:4627 , piece_id=GT4K2RYPOKWXA62QTYDFUVGQC7MMRWXXKB5ZJ6YQIKGKT3SWZEUA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.749723326 (887h51m44.620806371s) [noise=true , address=145.79.227.17:4972 , piece_id=WWFXI2QSE25XAV3LIXK26YMZURW3HJ7PJWVCZPU6Q3MJ7ZU4AVHA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.750167912 (887h51m44.621250947s) [noise=true , address=145.79.227.41:4085 , piece_id=CB27PJDJOIHXV2JLVVOY2JXXDZ7HB6JQZTH6YGYNIVTZQMTP4CJQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.750465619 (887h51m44.621548664s) [noise=true , address=145.79.227.13:4804 , piece_id=Z3H3KFSARJMZDAIWXZ7X52ERD73PHQ4EQTBAF7TH4LX6VHPDDLEQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.750839451 (887h51m44.621922486s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.751162165 (887h51m44.62224521s) [noise=true , address=145.79.227.5:4223 , piece_id=6RMRBZNGRFTUMGRDOJANB5CIBYJRU4G5EHZOFSMDSX6UNIKINJHQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.751499277 (887h51m44.622582312s) [noise=true , address=145.79.227.18:4006 , piece_id=RDSUXZAJXHDSQGNZ4BSOIGZYK2DBAOS3OKCXD7EKUXGLVB7SXFVQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.751801051 (887h51m44.622884086s) [noise=true , address=145.79.227.11:4729 , piece_id=OUQFHEARVMKMD3NOSYNK3SGS65GHO2VVWI33TMDSGNZSVI6U2GGQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.752107515 (887h51m44.62319056s) [noise=true , address=45.91.116.13:4156 , piece_id=ZSQICLD2EZCVFCWHWBBPHLBJQJ5EAIYQG2MBQWACDPAX5RDPKXOQ, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.752505893 (887h51m44.623588938s) [noise=true , address=145.79.227.19:4079 , piece_id=QMKDT3RBPNHPJMUIAJAFPEAYT4YJ7ECHCSGZWDNSBMSSJR34NUFA, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:44.753696261 (887h51m44.624779306s) [noise=true , address=145.79.227.40:4051 , piece_id=3S7ANT2SY7UNVUGT76H4SM3OIC5CY6JTTE2LR3WVOIYWA6IIWB3A, seg_pos={0 0}, upload=776950]: Uploading piece... Jan 11 15:20:47.390523966 (887h51m47.261607001s) [upload=776950]: Next returned nil segment Jan 11 15:20:47.390543012 (887h51m47.261626057s) [upload=776950]: Waiting for error group managing segments... Jan 11 15:20:47.822599671 (887h51m47.693682706s) [noise=true , address=145.79.227.19:4079 , piece_id=QMKDT3RBPNHPJMUIAJAFPEAYT4YJ7ECHCSGZWDNSBMSSJR34NUFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:47.998445353 (887h51m47.869528388s) [noise=true , address=45.91.116.13:4156 , piece_id=ZSQICLD2EZCVFCWHWBBPHLBJQJ5EAIYQG2MBQWACDPAX5RDPKXOQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:48.230600154 (887h51m48.101683199s) [noise=true , address=77.90.31.13:4627 , piece_id=GT4K2RYPOKWXA62QTYDFUVGQC7MMRWXXKB5ZJ6YQIKGKT3SWZEUA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.569294804 (887h51m49.440377839s) [noise=true , address=145.79.227.13:4804 , piece_id=Z3H3KFSARJMZDAIWXZ7X52ERD73PHQ4EQTBAF7TH4LX6VHPDDLEQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.591086838 (887h51m49.462169873s) [noise=true , address=145.79.227.12:4765 , piece_id=JIZAWBTRKZKBBHLTXFURJOBJXLAHF34MYA6ZCJWDTZKEWTKHSA7Q, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.733493227 (887h51m49.604576272s) [noise=true , address=145.79.227.41:4085 , piece_id=CB27PJDJOIHXV2JLVVOY2JXXDZ7HB6JQZTH6YGYNIVTZQMTP4CJQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.794463392 (887h51m49.665546427s) [noise=true , address=145.79.227.17:4972 , piece_id=WWFXI2QSE25XAV3LIXK26YMZURW3HJ7PJWVCZPU6Q3MJ7ZU4AVHA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.967238577 (887h51m49.838321632s) [noise=true , address=145.79.227.18:4006 , piece_id=RDSUXZAJXHDSQGNZ4BSOIGZYK2DBAOS3OKCXD7EKUXGLVB7SXFVQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:49.989894235 (887h51m49.86097727s) [noise=true , address=145.79.227.22:4509 , piece_id=SAASLZ536HTQD5SSEK6XCLZLQHBQMU7XS5634662DDLIOK4SP4IQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:50.725655336 (887h51m50.596738371s) [noise=true , address=145.79.227.40:4051 , piece_id=3S7ANT2SY7UNVUGT76H4SM3OIC5CY6JTTE2LR3WVOIYWA6IIWB3A, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:53.061473450 (887h51m52.932556485s) [noise=true , address=145.79.227.5:4223 , piece_id=6RMRBZNGRFTUMGRDOJANB5CIBYJRU4G5EHZOFSMDSX6UNIKINJHQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:53.664200566 (887h51m53.535283601s) [noise=true , address=145.79.227.33:4869 , piece_id=CH2RKQPDTDMB4XC4GT6I22SLZWSGZYCAUBSP3EAGN66KNNKODEFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:54.439611625 (887h51m54.31069466s) [noise=true , address=145.79.227.11:4729 , piece_id=OUQFHEARVMKMD3NOSYNK3SGS65GHO2VVWI33TMDSGNZSVI6U2GGQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: <nil> Jan 11 15:20:54.439630861 (887h51m54.310713896s) [seg_pos={0 0}, upload=776950]: Segment reached optimal threshold of 13 pieces. Jan 11 15:20:54.439937875 (887h51m54.311020911s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF Jan 11 15:20:54.439980266 (887h51m54.311063311s) [noise=true , address=145.79.227.4:4186 , piece_id=Y6CSHB7Y34DHRPC7LUOOU6XTR7YBFN55AO3WACBP4PZ27W6SHYDQ, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold Jan 11 15:20:54.439998441 (887h51m54.311081476s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: read tcp 188.166.225.170:41126->145.79.227.49:4337: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 15:20:54.440006246 (887h51m54.311089281s) [noise=true , address=145.79.227.49:4337 , piece_id=7WDGGLMPMDKUQFE3HQMQLSOHDBEQBGDY7TOLLO5C57WQNUOS6XYQ, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold Jan 11 15:20:54.440191488 (887h51m54.311274523s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1): protocol: read tcp 188.166.225.170:36086->145.79.227.24:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 15:20:54.440199785 (887h51m54.31128282s) [noise=true , address=145.79.227.24:4581 , piece_id=VWRERMH6WSK62ROW2MTJT3SXXJHFU4DG27OQLVQBTEN6Q4XIKTFA, seg_pos={0 0}, upload=776950]: Long tail cancellation due to optimal threshold Jan 11 15:20:54.440216286 (887h51m54.311299321s) [seg_pos={0 0}, upload=776950]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 15:20:54.440321306 (887h51m54.311404341s) [upload=776950]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 15:20:54.440326495 (887h51m54.31140953s) [upload=776950]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 15:20:54.726699085 (887h51m54.59778212s) [upload=776950]: Done uploading segments... Jan 11 15:20:54.726719413 (887h51m54.597802458s) [upload=776950]: Upload finished. err: <nil> Duration: 5.879714638s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 1.976596361s trace: https://jaeger.dev.storj.io/trace/c31ec8b6bb524c FAIL (2026-01-11 15:35:49 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.976622772s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [XXXXXXXXXXX.XXXXXXXXX.X.XX.XX...X..XXX......XXXXX...X.XXXXXX] duration: 30.000820759s trace: https://jaeger.dev.storj.io/trace/3e922f877e9b11b6 FAIL (2026-01-11 15:35:41 UTC) orbiter warning: context deadline exceeded Duration: 30.000843143s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.893842044s trace: https://jaeger.dev.storj.io/trace/db08c1a98355252 FAIL (2026-01-11 15:35:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, Piece ID: D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA) context deadline exceeded; piecestore: (Node ID: 1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, Piece ID: IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ) context deadline exceeded; piecestore: (Node ID: 12iDqXUpUqsDwhzKGNY1kMqEhS6vbHiESooVR9xu92B3SGybNmn, Piece ID: 3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ) context deadline exceeded ======LOGS====== =ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.515238933 (887h51m9.386321968s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.515935319 (887h51m9.387018354s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.516392410 (887h51m9.387475455s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.516885660 (887h51m9.387968705s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.517268609 (887h51m9.388351654s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.517740817 (887h51m9.388823852s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.518341942 (887h51m9.389424977s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.518779846 (887h51m9.389862881s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.519286630 (887h51m9.390369676s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.521517579 (887h51m9.392600614s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.524918765 (887h51m9.39600181s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.527183728 (887h51m9.398266783s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.529232859 (887h51m9.400315894s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.532984954 (887h51m9.404067999s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.536582354 (887h51m9.407665399s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:09.539559423 (887h51m9.410642468s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Uploading piece... Jan 11 15:20:10.837257700 (887h51m10.708340735s) [upload=776945]: Next returned nil segment Jan 11 15:20:10.837276767 (887h51m10.708359812s) [upload=776945]: Waiting for error group managing segments... Jan 11 15:20:11.321612300 (887h51m11.192695346s) [noise=true , address=64.49.13.169:4705 , piece_id=MXUAMIGG3TCY4TRJXDU4CRM5KZFLX2IQGIWVUAW7Z7URM5DMH34A, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:12.321151387 (887h51m12.192234433s) [noise=true , address=193.149.164.143:4694 , piece_id=AE6GGA7VRBUQDB7KGPDEM5YJOVWGEUFL7DAUUQN62HIBNK5GO5NA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:12.481216511 (887h51m12.352299546s) [noise=true , address=145.79.227.34:4926 , piece_id=45C263OTJH4KZG5QYWYCKORYWBO3OEZIJ7UIM6DO2LPCOCJJBVTA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:12.632251410 (887h51m12.503334445s) [noise=true , address=45.91.116.5:4320 , piece_id=3WCQLLVUYVJ7R7WLKXKY2HL5NG3Z2TQO5JH33AWAS6HPJRJZ3ZLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.302406869 (887h51m13.173489904s) [noise=true , address=77.90.31.15:4729 , piece_id=IX2NLKVP7PXNQEW2J6C6YQIS3N4GYAW7YLOJIAMZW3T7VKTCXHDQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.417486975 (887h51m13.28857001s) [noise=true , address=77.90.31.21:4902 , piece_id=6FH5QXSO5R7CE4UFSFOZAJAKPT7PLPJ67ULH5CHSOS2TD5HPQNEA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.437898801 (887h51m13.308981846s) [noise=true , address=145.79.227.33:4897 , piece_id=ELAEQQCOWHEYLLO6XBSL5N5FZWDJXRLSYFB5HRIK4WFAA7SEBNLA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.666780260 (887h51m13.537863305s) [noise=true , address=145.79.227.40:4078 , piece_id=S66ITLEVFPMH4BGNVLRJMWWFVVXABDW264OLPXAD3P5CKL3IPM4Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:13.677746121 (887h51m13.548829156s) [noise=true , address=45.91.116.10:4104 , piece_id=NCG7WEDV7A4E5H4D3WGWI2DMEUSFFPU7YYXLFJDNJQAARSMPBW2Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:14.339206281 (887h51m14.210289316s) [noise=true , address=145.79.227.24:4605 , piece_id=D5ZMAB7PNP4CFTYRJGIMW7MQ5KMOCOHYXDWUHH4MWASDT6QZFOJA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:14.557434685 (887h51m14.42851773s) [noise=true , address=45.91.116.4:4969 , piece_id=LK6YTWXVI3O3GWI3TZKBIJWEJZQBGZW76MSO4TASXWBPGQLMJNHQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:15.047052989 (887h51m14.918136024s) [noise=true , address=77.90.31.11:4574 , piece_id=D6H557N23KO5L32TEYGYNSY7YHRPTW4VSN56ZLQYYAI5R6RN3FSA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:15.347734503 (887h51m15.218817548s) [noise=true , address=145.79.227.44:4209 , piece_id=LPWQWW75RWZM7SZYTZOPGDZ6UMN3SMQSC3NA47GHAMX2KBVT32UA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:15.431090403 (887h51m15.302173448s) [noise=true , address=145.79.227.43:4190 , piece_id=FVDGEK6UTD4M7XFSVFE3CLAJSADDOTKNPLW3F6FH6UVRWR6KS66Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: <nil> Jan 11 15:20:15.431113807 (887h51m15.302196852s) [seg_pos={0 0}, upload=776945]: Segment reached optimal threshold of 14 pieces. Jan 11 15:20:15.431355577 (887h51m15.302438622s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cCPRmsdkrVKFyMfLdBNYhnDhcA2Los7kbvuYUT6dcofPLVXZ7): protocol: expected piece hash; context canceled; EOF Jan 11 15:20:15.431362331 (887h51m15.302445376s) [noise=true , address=45.91.116.11:4134 , piece_id=LPBJC6HNWKDY242AW7PADHZN4E25UPPZW7L27E3BAYCAUWORMJLQ, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold Jan 11 15:20:15.431472140 (887h51m15.302555175s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Jan 11 15:20:15.431491286 (887h51m15.302574321s) [noise=true , address=77.90.31.19:4858 , piece_id=VKVEGCO4SSCWV5KB4WMPQ3V5S6ZUKEK4L5IKUTOG3CXUJO32QPHA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold Jan 11 15:20:15.431505644 (887h51m15.302588679s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9): protocol: expected piece hash; context canceled; EOF Jan 11 15:20:15.431513218 (887h51m15.302596263s) [noise=true , address=45.91.116.14:4222 , piece_id=TKH3NPJHUV2CXLLFVUEHMLWTYA2AHKTRRX73OGVJF376BUTDOK6Q, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold Jan 11 15:20:15.431640601 (887h51m15.302723646s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: read tcp 188.166.225.170:45042->145.79.227.30:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 15:20:15.431654497 (887h51m15.302737572s) [noise=true , address=145.79.227.30:4784 , piece_id=VNIGF7FXZDPCUUJM723BUVG3A5D7T5G6WUSRPA2TZTR245ZKGNDA, seg_pos={0 0}, upload=776945]: Long tail cancellation due to optimal threshold Jan 11 15:20:15.431749248 (887h51m15.302832283s) [seg_pos={0 0}, upload=776945]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 11 15:20:15.431840110 (887h51m15.302923155s) [upload=776945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 15:20:15.431847053 (887h51m15.302930088s) [upload=776945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 15:20:15.728771354 (887h51m15.599854419s) [upload=776945]: Done uploading segments... Jan 11 15:20:15.728795108 (887h51m15.599878153s) [upload=776945]: Upload finished. err: <nil> Duration: 5.893874244s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [XXXXXXX.X.XXXXXXXXXXXX.XXXXX.XXXXXXXXXX.XX.XXXXX.XXXXX.X.XXX] duration: 5.000705981s trace: https://jaeger.dev.storj.io/trace/2aba73bfa7cea6cb FAIL (2026-01-11 15:35:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000722724s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [X....XX.X..X......X.X.....................X........X......X.] duration: 28.461174208s trace: https://jaeger.dev.storj.io/trace/174ef70bce9271a3 FAIL (2026-01-11 15:35:21 UTC) orbiter warning: context deadline exceeded Duration: 28.461189017s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.XXXXXXXXX.X.XX.XX...X..XXX......XXXXX...X.XXXXX] duration: 30.001399901s trace: https://jaeger.dev.storj.io/trace/73707e3575515f2d FAIL (2026-01-11 15:35:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001412013s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [XXXXXXXX.X.XXXXXXXXXXXX.XXXXX.XXXXXXXXXX.XX.XXXXX.XXXXX.X.XX] duration: 5.000744113s trace: https://jaeger.dev.storj.io/trace/731b78e80ae07e87 FAIL (2026-01-11 15:35:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000760135s core uplink us-select-1 us1-placement-20

1
...
392
393
394
395
396
...
9048