Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 1.344982277s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 11:56:08 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.345011084s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [......XXX.XX......X.X..XX.X.X...X....X......X.X.X...X.X.....] duration: 5.692314377s trace: https://jaeger.dev.storj.io/trace/28e3ccf01bc9e478 FAIL (2025-12-26 11:55:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: IZRHL6SKMFALQ75VLSR6ZBGNSQKSH6F4RZB7KXW5RWI5G7KNR5YA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: WKISHW5AVXBJPXICXH3LVTFAB4DO72INFTRWP67JTGLL4HBSPQUA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 7L2XKXGUBTS763VQUC5AYRR27YTJOW4BJTYVOHN24H54JZC7LKZQ) context deadline exceeded ======LOGS====== Dec 26 11:51:45.806103832 (500h22m45.677186887s) [upload=439163]: Starting upload Dec 26 11:51:45.806222699 (500h22m45.677305754s) [upload=439163]: Done starting upload Dec 26 11:51:45.806269387 (500h22m45.677352463s) [upload=439163]: Uploading segments... Dec 26 11:51:45.806469509 (500h22m45.677552554s) [upload=439163]: Got next segment. Inline: false Dec 26 11:51:45.806510096 (500h22m45.677593141s) [upload=439163]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:51:45.806516167 (500h22m45.677599212s) [upload=439163]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:51:46.053749784 (500h22m45.92483283s) [seg_pos={0 0}, upload=439163]: Begin upload segment... Dec 26 11:51:46.053825158 (500h22m45.924908203s) [seg_pos={0 0}, upload=439163]: Done begin upload segment. Dec 26 11:51:46.053853322 (500h22m45.924936357s) [noise=true , address=145.79.227.33:4870 , piece_id=GJUKOF3IYXKWYTAJDONYXYR5H2QFOWSMJ5PFP2F6QBQ4ZMB74ZGQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.053898177 (500h22m45.924981242s) [noise=true , address=45.91.116.4:4937 , piece_id=N2LTAF2SR7LSDVAGLK2KURMN5BSK6HURPDEXL4WXGZMZXAR5DAMQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.054859197 (500h22m45.925942242s) [noise=true , address=145.79.227.31:4799 , piece_id=KLIQEQJ67N3NTFHF4GNAW7MEUEJ22YZOWNCIZASR7ADELUA4UJ5Q, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.055169739 (500h22m45.926252784s) [noise=true , address=145.79.227.12:4765 , piece_id=DFUA3GX2F5WFZRHWWKRWDWNLK4KYVEEZMMNLYMOLKMUNOWNVGZMQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.055444532 (500h22m45.926527567s) [noise=true , address=145.79.227.15:4875 , piece_id=FGC37BWAWWLNS6FX2RDPCDJ2FCMGY7E5JMCB7QNRT3Q2ENIBQ5EA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.055695850 (500h22m45.926778895s) [noise=true , address=145.79.227.11:4732 , piece_id=SBNRQTRG2YRJH7Y7SD3UX5TI3AFLUCVCXBLJL65PVSAHKIM2RLAQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.056146729 (500h22m45.927229764s) [noise=true , address=145.79.227.49:4337 , piece_id=NCNSJ5VWGAMZ4IPZCSK7T4PCGKC4IJHUNIRSQGHXUXH433KDD5KQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.056454064 (500h22m45.927537099s) [noise=true , address=145.79.227.52:4446 , piece_id=CQ5ZYB5QYRENMRV4MP76KYIJECIU2HTW7IO2QWOULX77DYETYRRA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.057571462 (500h22m45.928654507s) [noise=true , address=45.91.116.11:4118 , piece_id=WKISHW5AVXBJPXICXH3LVTFAB4DO72INFTRWP67JTGLL4HBSPQUA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.058400641 (500h22m45.929483686s) [noise=true , address=145.79.227.51:4410 , piece_id=BQLHMZT7ZRBOFPVDUMY3IL42IPJQ6E3FSDWYSSI3HHB6J6MUISYQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.058824157 (500h22m45.929907202s) [noise=true , address=145.79.227.46:4603 , piece_id=MD3G53QYNDJVYMJ6ERS5E5GM5R7V55XEAWKMYJGXKCVUG7KADGZQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.058936591 (500h22m45.930019626s) [noise=true , address=45.91.116.9:4262 , piece_id=67TKAHQYFTJSF5WIOW6VAPOCQ66ECKQLL34OXSKBZHLWY4FKR4VA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.059430712 (500h22m45.930513747s) [noise=true , address=145.79.227.17:4970 , piece_id=2Q77VM35EUEUDHR47FGAPCBBHEX3N3QFGOGY5GF6PNEFP6ZMCU2Q, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.059797260 (500h22m45.930880295s) [noise=true , address=77.90.31.13:4627 , piece_id=7L2XKXGUBTS763VQUC5AYRR27YTJOW4BJTYVOHN24H54JZC7LKZQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.060238070 (500h22m45.931321105s) [noise=true , address=77.90.31.10:4519 , piece_id=IZRHL6SKMFALQ75VLSR6ZBGNSQKSH6F4RZB7KXW5RWI5G7KNR5YA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.060510879 (500h22m45.931593924s) [noise=true , address=145.79.227.44:4193 , piece_id=JXRRRGDODZYWSDUJM4WX6RTZ2TWABCFDYFO6RKBWRFICXLAG6TVQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:47.674430101 (500h22m47.545513146s) [upload=439163]: Next returned nil segment Dec 26 11:51:47.674444337 (500h22m47.545527372s) [upload=439163]: Waiting for error group managing segments... Dec 26 11:51:48.058568168 (500h22m47.929651203s) [noise=true , address=145.79.227.46:4603 , piece_id=MD3G53QYNDJVYMJ6ERS5E5GM5R7V55XEAWKMYJGXKCVUG7KADGZQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.058901202 (500h22m47.929984237s) [noise=true , address=145.79.227.12:4765 , piece_id=DFUA3GX2F5WFZRHWWKRWDWNLK4KYVEEZMMNLYMOLKMUNOWNVGZMQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.060248368 (500h22m47.931331413s) [noise=true , address=145.79.227.15:4875 , piece_id=FGC37BWAWWLNS6FX2RDPCDJ2FCMGY7E5JMCB7QNRT3Q2ENIBQ5EA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.060436035 (500h22m47.93151908s) [noise=true , address=145.79.227.11:4732 , piece_id=SBNRQTRG2YRJH7Y7SD3UX5TI3AFLUCVCXBLJL65PVSAHKIM2RLAQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.063895081 (500h22m47.934978126s) [noise=true , address=145.79.227.49:4337 , piece_id=NCNSJ5VWGAMZ4IPZCSK7T4PCGKC4IJHUNIRSQGHXUXH433KDD5KQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.067448307 (500h22m47.938531352s) [noise=true , address=145.79.227.17:4970 , piece_id=2Q77VM35EUEUDHR47FGAPCBBHEX3N3QFGOGY5GF6PNEFP6ZMCU2Q, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.067687342 (500h22m47.938770377s) [noise=true , address=145.79.227.31:4799 , piece_id=KLIQEQJ67N3NTFHF4GNAW7MEUEJ22YZOWNCIZASR7ADELUA4UJ5Q, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.069843568 (500h22m47.940926603s) [noise=true , address=145.79.227.52:4446 , piece_id=CQ5ZYB5QYRENMRV4MP76KYIJECIU2HTW7IO2QWOULX77DYETYRRA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.079236143 (500h22m47.950319189s) [noise=true , address=145.79.227.51:4410 , piece_id=BQLHMZT7ZRBOFPVDUMY3IL42IPJQ6E3FSDWYSSI3HHB6J6MUISYQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.273171081 (500h22m48.144254126s) [noise=true , address=77.90.31.10:4519 , piece_id=IZRHL6SKMFALQ75VLSR6ZBGNSQKSH6F4RZB7KXW5RWI5G7KNR5YA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.337012426 (500h22m48.208095471s) [noise=true , address=45.91.116.11:4118 , piece_id=WKISHW5AVXBJPXICXH3LVTFAB4DO72INFTRWP67JTGLL4HBSPQUA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.655203850 (500h22m48.526286895s) [noise=true , address=45.91.116.4:4937 , piece_id=N2LTAF2SR7LSDVAGLK2KURMN5BSK6HURPDEXL4WXGZMZXAR5DAMQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:49.230200346 (500h22m49.101283381s) [noise=true , address=77.90.31.13:4627 , piece_id=7L2XKXGUBTS763VQUC5AYRR27YTJOW4BJTYVOHN24H54JZC7LKZQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:49.230227107 (500h22m49.101310152s) [seg_pos={0 0}, upload=439163]: Segment reached optimal threshold of 13 pieces. Dec 26 11:51:49.230425084 (500h22m49.101508129s) [noise=true , address=145.79.227.33:4870 , piece_id=GJUKOF3IYXKWYTAJDONYXYR5H2QFOWSMJ5PFP2F6QBQ4ZMB74ZGQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 11:51:49.230432528 (500h22m49.101515563s) [noise=true , address=145.79.227.33:4870 , piece_id=GJUKOF3IYXKWYTAJDONYXYR5H2QFOWSMJ5PFP2F6QBQ4ZMB74ZGQ, seg_pos={0 0}, upload=439163]: Long tail cancellation due to optimal threshold Dec 26 11:51:49.230576362 (500h22m49.101659407s) [noise=true , address=145.79.227.44:4193 , piece_id=JXRRRGDODZYWSDUJM4WX6RTZ2TWABCFDYFO6RKBWRFICXLAG6TVQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 11:51:49.230583695 (500h22m49.101666731s) [noise=true , address=145.79.227.44:4193 , piece_id=JXRRRGDODZYWSDUJM4WX6RTZ2TWABCFDYFO6RKBWRFICXLAG6TVQ, seg_pos={0 0}, upload=439163]: Long tail cancellation due to optimal threshold Dec 26 11:51:49.230812872 (500h22m49.101895907s) [noise=true , address=45.91.116.9:4262 , piece_id=67TKAHQYFTJSF5WIOW6VAPOCQ66ECKQLL34OXSKBZHLWY4FKR4VA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 11:51:49.230819655 (500h22m49.1019027s) [noise=true , address=45.91.116.9:4262 , piece_id=67TKAHQYFTJSF5WIOW6VAPOCQ66ECKQLL34OXSKBZHLWY4FKR4VA, seg_pos={0 0}, upload=439163]: Long tail cancellation due to optimal threshold Dec 26 11:51:49.230836246 (500h22m49.101919291s) [seg_pos={0 0}, upload=439163]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:51:49.230905047 (500h22m49.101988092s) [upload=439163]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:51:49.230909446 (500h22m49.101992481s) [upload=439163]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:51:49.530474351 (500h22m49.401557396s) [upload=439163]: Done uploading segments... Dec 26 11:51:49.530495973 (500h22m49.401579018s) [upload=439163]: Upload finished. err: <nil> Duration: 5.692341118s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 1.33358786s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 11:55:46 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.333632005s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........................................X....X............X] duration: 5.00127825s trace: https://jaeger.dev.storj.io/trace/1f892730a4501602 FAIL (2025-12-26 11:55:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001301252s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X......XXX.XX......X.X..XX.X.X...X....X......X.X.X...X.X....] duration: 5.770359558s trace: https://jaeger.dev.storj.io/trace/28e3ccf01bc9e478 FAIL (2025-12-26 11:55:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: IZRHL6SKMFALQ75VLSR6ZBGNSQKSH6F4RZB7KXW5RWI5G7KNR5YA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: WKISHW5AVXBJPXICXH3LVTFAB4DO72INFTRWP67JTGLL4HBSPQUA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 7L2XKXGUBTS763VQUC5AYRR27YTJOW4BJTYVOHN24H54JZC7LKZQ) context deadline exceeded ======LOGS====== Dec 26 11:51:45.806103832 (500h22m45.677186887s) [upload=439163]: Starting upload Dec 26 11:51:45.806222699 (500h22m45.677305754s) [upload=439163]: Done starting upload Dec 26 11:51:45.806269387 (500h22m45.677352463s) [upload=439163]: Uploading segments... Dec 26 11:51:45.806469509 (500h22m45.677552554s) [upload=439163]: Got next segment. Inline: false Dec 26 11:51:45.806510096 (500h22m45.677593141s) [upload=439163]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:51:45.806516167 (500h22m45.677599212s) [upload=439163]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:51:46.053749784 (500h22m45.92483283s) [seg_pos={0 0}, upload=439163]: Begin upload segment... Dec 26 11:51:46.053825158 (500h22m45.924908203s) [seg_pos={0 0}, upload=439163]: Done begin upload segment. Dec 26 11:51:46.053853322 (500h22m45.924936357s) [noise=true , address=145.79.227.33:4870 , piece_id=GJUKOF3IYXKWYTAJDONYXYR5H2QFOWSMJ5PFP2F6QBQ4ZMB74ZGQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.053898177 (500h22m45.924981242s) [noise=true , address=45.91.116.4:4937 , piece_id=N2LTAF2SR7LSDVAGLK2KURMN5BSK6HURPDEXL4WXGZMZXAR5DAMQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.054859197 (500h22m45.925942242s) [noise=true , address=145.79.227.31:4799 , piece_id=KLIQEQJ67N3NTFHF4GNAW7MEUEJ22YZOWNCIZASR7ADELUA4UJ5Q, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.055169739 (500h22m45.926252784s) [noise=true , address=145.79.227.12:4765 , piece_id=DFUA3GX2F5WFZRHWWKRWDWNLK4KYVEEZMMNLYMOLKMUNOWNVGZMQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.055444532 (500h22m45.926527567s) [noise=true , address=145.79.227.15:4875 , piece_id=FGC37BWAWWLNS6FX2RDPCDJ2FCMGY7E5JMCB7QNRT3Q2ENIBQ5EA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.055695850 (500h22m45.926778895s) [noise=true , address=145.79.227.11:4732 , piece_id=SBNRQTRG2YRJH7Y7SD3UX5TI3AFLUCVCXBLJL65PVSAHKIM2RLAQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.056146729 (500h22m45.927229764s) [noise=true , address=145.79.227.49:4337 , piece_id=NCNSJ5VWGAMZ4IPZCSK7T4PCGKC4IJHUNIRSQGHXUXH433KDD5KQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.056454064 (500h22m45.927537099s) [noise=true , address=145.79.227.52:4446 , piece_id=CQ5ZYB5QYRENMRV4MP76KYIJECIU2HTW7IO2QWOULX77DYETYRRA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.057571462 (500h22m45.928654507s) [noise=true , address=45.91.116.11:4118 , piece_id=WKISHW5AVXBJPXICXH3LVTFAB4DO72INFTRWP67JTGLL4HBSPQUA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.058400641 (500h22m45.929483686s) [noise=true , address=145.79.227.51:4410 , piece_id=BQLHMZT7ZRBOFPVDUMY3IL42IPJQ6E3FSDWYSSI3HHB6J6MUISYQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.058824157 (500h22m45.929907202s) [noise=true , address=145.79.227.46:4603 , piece_id=MD3G53QYNDJVYMJ6ERS5E5GM5R7V55XEAWKMYJGXKCVUG7KADGZQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.058936591 (500h22m45.930019626s) [noise=true , address=45.91.116.9:4262 , piece_id=67TKAHQYFTJSF5WIOW6VAPOCQ66ECKQLL34OXSKBZHLWY4FKR4VA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.059430712 (500h22m45.930513747s) [noise=true , address=145.79.227.17:4970 , piece_id=2Q77VM35EUEUDHR47FGAPCBBHEX3N3QFGOGY5GF6PNEFP6ZMCU2Q, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.059797260 (500h22m45.930880295s) [noise=true , address=77.90.31.13:4627 , piece_id=7L2XKXGUBTS763VQUC5AYRR27YTJOW4BJTYVOHN24H54JZC7LKZQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.060238070 (500h22m45.931321105s) [noise=true , address=77.90.31.10:4519 , piece_id=IZRHL6SKMFALQ75VLSR6ZBGNSQKSH6F4RZB7KXW5RWI5G7KNR5YA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.060510879 (500h22m45.931593924s) [noise=true , address=145.79.227.44:4193 , piece_id=JXRRRGDODZYWSDUJM4WX6RTZ2TWABCFDYFO6RKBWRFICXLAG6TVQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:47.674430101 (500h22m47.545513146s) [upload=439163]: Next returned nil segment Dec 26 11:51:47.674444337 (500h22m47.545527372s) [upload=439163]: Waiting for error group managing segments... Dec 26 11:51:48.058568168 (500h22m47.929651203s) [noise=true , address=145.79.227.46:4603 , piece_id=MD3G53QYNDJVYMJ6ERS5E5GM5R7V55XEAWKMYJGXKCVUG7KADGZQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.058901202 (500h22m47.929984237s) [noise=true , address=145.79.227.12:4765 , piece_id=DFUA3GX2F5WFZRHWWKRWDWNLK4KYVEEZMMNLYMOLKMUNOWNVGZMQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.060248368 (500h22m47.931331413s) [noise=true , address=145.79.227.15:4875 , piece_id=FGC37BWAWWLNS6FX2RDPCDJ2FCMGY7E5JMCB7QNRT3Q2ENIBQ5EA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.060436035 (500h22m47.93151908s) [noise=true , address=145.79.227.11:4732 , piece_id=SBNRQTRG2YRJH7Y7SD3UX5TI3AFLUCVCXBLJL65PVSAHKIM2RLAQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.063895081 (500h22m47.934978126s) [noise=true , address=145.79.227.49:4337 , piece_id=NCNSJ5VWGAMZ4IPZCSK7T4PCGKC4IJHUNIRSQGHXUXH433KDD5KQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.067448307 (500h22m47.938531352s) [noise=true , address=145.79.227.17:4970 , piece_id=2Q77VM35EUEUDHR47FGAPCBBHEX3N3QFGOGY5GF6PNEFP6ZMCU2Q, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.067687342 (500h22m47.938770377s) [noise=true , address=145.79.227.31:4799 , piece_id=KLIQEQJ67N3NTFHF4GNAW7MEUEJ22YZOWNCIZASR7ADELUA4UJ5Q, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.069843568 (500h22m47.940926603s) [noise=true , address=145.79.227.52:4446 , piece_id=CQ5ZYB5QYRENMRV4MP76KYIJECIU2HTW7IO2QWOULX77DYETYRRA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.079236143 (500h22m47.950319189s) [noise=true , address=145.79.227.51:4410 , piece_id=BQLHMZT7ZRBOFPVDUMY3IL42IPJQ6E3FSDWYSSI3HHB6J6MUISYQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.273171081 (500h22m48.144254126s) [noise=true , address=77.90.31.10:4519 , piece_id=IZRHL6SKMFALQ75VLSR6ZBGNSQKSH6F4RZB7KXW5RWI5G7KNR5YA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.337012426 (500h22m48.208095471s) [noise=true , address=45.91.116.11:4118 , piece_id=WKISHW5AVXBJPXICXH3LVTFAB4DO72INFTRWP67JTGLL4HBSPQUA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.655203850 (500h22m48.526286895s) [noise=true , address=45.91.116.4:4937 , piece_id=N2LTAF2SR7LSDVAGLK2KURMN5BSK6HURPDEXL4WXGZMZXAR5DAMQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:49.230200346 (500h22m49.101283381s) [noise=true , address=77.90.31.13:4627 , piece_id=7L2XKXGUBTS763VQUC5AYRR27YTJOW4BJTYVOHN24H54JZC7LKZQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:49.230227107 (500h22m49.101310152s) [seg_pos={0 0}, upload=439163]: Segment reached optimal threshold of 13 pieces. Dec 26 11:51:49.230425084 (500h22m49.101508129s) [noise=true , address=145.79.227.33:4870 , piece_id=GJUKOF3IYXKWYTAJDONYXYR5H2QFOWSMJ5PFP2F6QBQ4ZMB74ZGQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 11:51:49.230432528 (500h22m49.101515563s) [noise=true , address=145.79.227.33:4870 , piece_id=GJUKOF3IYXKWYTAJDONYXYR5H2QFOWSMJ5PFP2F6QBQ4ZMB74ZGQ, seg_pos={0 0}, upload=439163]: Long tail cancellation due to optimal threshold Dec 26 11:51:49.230576362 (500h22m49.101659407s) [noise=true , address=145.79.227.44:4193 , piece_id=JXRRRGDODZYWSDUJM4WX6RTZ2TWABCFDYFO6RKBWRFICXLAG6TVQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 11:51:49.230583695 (500h22m49.101666731s) [noise=true , address=145.79.227.44:4193 , piece_id=JXRRRGDODZYWSDUJM4WX6RTZ2TWABCFDYFO6RKBWRFICXLAG6TVQ, seg_pos={0 0}, upload=439163]: Long tail cancellation due to optimal threshold Dec 26 11:51:49.230812872 (500h22m49.101895907s) [noise=true , address=45.91.116.9:4262 , piece_id=67TKAHQYFTJSF5WIOW6VAPOCQ66ECKQLL34OXSKBZHLWY4FKR4VA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 11:51:49.230819655 (500h22m49.1019027s) [noise=true , address=45.91.116.9:4262 , piece_id=67TKAHQYFTJSF5WIOW6VAPOCQ66ECKQLL34OXSKBZHLWY4FKR4VA, seg_pos={0 0}, upload=439163]: Long tail cancellation due to optimal threshold Dec 26 11:51:49.230836246 (500h22m49.101919291s) [seg_pos={0 0}, upload=439163]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:51:49.230905047 (500h22m49.101988092s) [upload=439163]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:51:49.230909446 (500h22m49.101992481s) [upload=439163]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:51:49.530474351 (500h22m49.401557396s) [upload=439163]: Done uploading segments... Dec 26 11:51:49.530495973 (500h22m49.401579018s) [upload=439163]: Upload finished. err: <nil> Duration: 5.770400576s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 1.408611017s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 11:55:09 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.40863818s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 1.359668683s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 11:54:48 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.35970404s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XX......XXX.XX......X.X..XX.X.X...X....X......X.X.X...X.X...] duration: 5.865899052s trace: https://jaeger.dev.storj.io/trace/28e3ccf01bc9e478 FAIL (2025-12-26 11:54:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: IZRHL6SKMFALQ75VLSR6ZBGNSQKSH6F4RZB7KXW5RWI5G7KNR5YA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: WKISHW5AVXBJPXICXH3LVTFAB4DO72INFTRWP67JTGLL4HBSPQUA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 7L2XKXGUBTS763VQUC5AYRR27YTJOW4BJTYVOHN24H54JZC7LKZQ) context deadline exceeded ======LOGS====== Dec 26 11:51:45.806103832 (500h22m45.677186887s) [upload=439163]: Starting upload Dec 26 11:51:45.806222699 (500h22m45.677305754s) [upload=439163]: Done starting upload Dec 26 11:51:45.806269387 (500h22m45.677352463s) [upload=439163]: Uploading segments... Dec 26 11:51:45.806469509 (500h22m45.677552554s) [upload=439163]: Got next segment. Inline: false Dec 26 11:51:45.806510096 (500h22m45.677593141s) [upload=439163]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:51:45.806516167 (500h22m45.677599212s) [upload=439163]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:51:46.053749784 (500h22m45.92483283s) [seg_pos={0 0}, upload=439163]: Begin upload segment... Dec 26 11:51:46.053825158 (500h22m45.924908203s) [seg_pos={0 0}, upload=439163]: Done begin upload segment. Dec 26 11:51:46.053853322 (500h22m45.924936357s) [noise=true , address=145.79.227.33:4870 , piece_id=GJUKOF3IYXKWYTAJDONYXYR5H2QFOWSMJ5PFP2F6QBQ4ZMB74ZGQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.053898177 (500h22m45.924981242s) [noise=true , address=45.91.116.4:4937 , piece_id=N2LTAF2SR7LSDVAGLK2KURMN5BSK6HURPDEXL4WXGZMZXAR5DAMQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.054859197 (500h22m45.925942242s) [noise=true , address=145.79.227.31:4799 , piece_id=KLIQEQJ67N3NTFHF4GNAW7MEUEJ22YZOWNCIZASR7ADELUA4UJ5Q, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.055169739 (500h22m45.926252784s) [noise=true , address=145.79.227.12:4765 , piece_id=DFUA3GX2F5WFZRHWWKRWDWNLK4KYVEEZMMNLYMOLKMUNOWNVGZMQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.055444532 (500h22m45.926527567s) [noise=true , address=145.79.227.15:4875 , piece_id=FGC37BWAWWLNS6FX2RDPCDJ2FCMGY7E5JMCB7QNRT3Q2ENIBQ5EA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.055695850 (500h22m45.926778895s) [noise=true , address=145.79.227.11:4732 , piece_id=SBNRQTRG2YRJH7Y7SD3UX5TI3AFLUCVCXBLJL65PVSAHKIM2RLAQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.056146729 (500h22m45.927229764s) [noise=true , address=145.79.227.49:4337 , piece_id=NCNSJ5VWGAMZ4IPZCSK7T4PCGKC4IJHUNIRSQGHXUXH433KDD5KQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.056454064 (500h22m45.927537099s) [noise=true , address=145.79.227.52:4446 , piece_id=CQ5ZYB5QYRENMRV4MP76KYIJECIU2HTW7IO2QWOULX77DYETYRRA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.057571462 (500h22m45.928654507s) [noise=true , address=45.91.116.11:4118 , piece_id=WKISHW5AVXBJPXICXH3LVTFAB4DO72INFTRWP67JTGLL4HBSPQUA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.058400641 (500h22m45.929483686s) [noise=true , address=145.79.227.51:4410 , piece_id=BQLHMZT7ZRBOFPVDUMY3IL42IPJQ6E3FSDWYSSI3HHB6J6MUISYQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.058824157 (500h22m45.929907202s) [noise=true , address=145.79.227.46:4603 , piece_id=MD3G53QYNDJVYMJ6ERS5E5GM5R7V55XEAWKMYJGXKCVUG7KADGZQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.058936591 (500h22m45.930019626s) [noise=true , address=45.91.116.9:4262 , piece_id=67TKAHQYFTJSF5WIOW6VAPOCQ66ECKQLL34OXSKBZHLWY4FKR4VA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.059430712 (500h22m45.930513747s) [noise=true , address=145.79.227.17:4970 , piece_id=2Q77VM35EUEUDHR47FGAPCBBHEX3N3QFGOGY5GF6PNEFP6ZMCU2Q, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.059797260 (500h22m45.930880295s) [noise=true , address=77.90.31.13:4627 , piece_id=7L2XKXGUBTS763VQUC5AYRR27YTJOW4BJTYVOHN24H54JZC7LKZQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.060238070 (500h22m45.931321105s) [noise=true , address=77.90.31.10:4519 , piece_id=IZRHL6SKMFALQ75VLSR6ZBGNSQKSH6F4RZB7KXW5RWI5G7KNR5YA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.060510879 (500h22m45.931593924s) [noise=true , address=145.79.227.44:4193 , piece_id=JXRRRGDODZYWSDUJM4WX6RTZ2TWABCFDYFO6RKBWRFICXLAG6TVQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:47.674430101 (500h22m47.545513146s) [upload=439163]: Next returned nil segment Dec 26 11:51:47.674444337 (500h22m47.545527372s) [upload=439163]: Waiting for error group managing segments... Dec 26 11:51:48.058568168 (500h22m47.929651203s) [noise=true , address=145.79.227.46:4603 , piece_id=MD3G53QYNDJVYMJ6ERS5E5GM5R7V55XEAWKMYJGXKCVUG7KADGZQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.058901202 (500h22m47.929984237s) [noise=true , address=145.79.227.12:4765 , piece_id=DFUA3GX2F5WFZRHWWKRWDWNLK4KYVEEZMMNLYMOLKMUNOWNVGZMQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.060248368 (500h22m47.931331413s) [noise=true , address=145.79.227.15:4875 , piece_id=FGC37BWAWWLNS6FX2RDPCDJ2FCMGY7E5JMCB7QNRT3Q2ENIBQ5EA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.060436035 (500h22m47.93151908s) [noise=true , address=145.79.227.11:4732 , piece_id=SBNRQTRG2YRJH7Y7SD3UX5TI3AFLUCVCXBLJL65PVSAHKIM2RLAQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.063895081 (500h22m47.934978126s) [noise=true , address=145.79.227.49:4337 , piece_id=NCNSJ5VWGAMZ4IPZCSK7T4PCGKC4IJHUNIRSQGHXUXH433KDD5KQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.067448307 (500h22m47.938531352s) [noise=true , address=145.79.227.17:4970 , piece_id=2Q77VM35EUEUDHR47FGAPCBBHEX3N3QFGOGY5GF6PNEFP6ZMCU2Q, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.067687342 (500h22m47.938770377s) [noise=true , address=145.79.227.31:4799 , piece_id=KLIQEQJ67N3NTFHF4GNAW7MEUEJ22YZOWNCIZASR7ADELUA4UJ5Q, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.069843568 (500h22m47.940926603s) [noise=true , address=145.79.227.52:4446 , piece_id=CQ5ZYB5QYRENMRV4MP76KYIJECIU2HTW7IO2QWOULX77DYETYRRA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.079236143 (500h22m47.950319189s) [noise=true , address=145.79.227.51:4410 , piece_id=BQLHMZT7ZRBOFPVDUMY3IL42IPJQ6E3FSDWYSSI3HHB6J6MUISYQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.273171081 (500h22m48.144254126s) [noise=true , address=77.90.31.10:4519 , piece_id=IZRHL6SKMFALQ75VLSR6ZBGNSQKSH6F4RZB7KXW5RWI5G7KNR5YA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.337012426 (500h22m48.208095471s) [noise=true , address=45.91.116.11:4118 , piece_id=WKISHW5AVXBJPXICXH3LVTFAB4DO72INFTRWP67JTGLL4HBSPQUA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.655203850 (500h22m48.526286895s) [noise=true , address=45.91.116.4:4937 , piece_id=N2LTAF2SR7LSDVAGLK2KURMN5BSK6HURPDEXL4WXGZMZXAR5DAMQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:49.230200346 (500h22m49.101283381s) [noise=true , address=77.90.31.13:4627 , piece_id=7L2XKXGUBTS763VQUC5AYRR27YTJOW4BJTYVOHN24H54JZC7LKZQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:49.230227107 (500h22m49.101310152s) [seg_pos={0 0}, upload=439163]: Segment reached optimal threshold of 13 pieces. Dec 26 11:51:49.230425084 (500h22m49.101508129s) [noise=true , address=145.79.227.33:4870 , piece_id=GJUKOF3IYXKWYTAJDONYXYR5H2QFOWSMJ5PFP2F6QBQ4ZMB74ZGQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 11:51:49.230432528 (500h22m49.101515563s) [noise=true , address=145.79.227.33:4870 , piece_id=GJUKOF3IYXKWYTAJDONYXYR5H2QFOWSMJ5PFP2F6QBQ4ZMB74ZGQ, seg_pos={0 0}, upload=439163]: Long tail cancellation due to optimal threshold Dec 26 11:51:49.230576362 (500h22m49.101659407s) [noise=true , address=145.79.227.44:4193 , piece_id=JXRRRGDODZYWSDUJM4WX6RTZ2TWABCFDYFO6RKBWRFICXLAG6TVQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 11:51:49.230583695 (500h22m49.101666731s) [noise=true , address=145.79.227.44:4193 , piece_id=JXRRRGDODZYWSDUJM4WX6RTZ2TWABCFDYFO6RKBWRFICXLAG6TVQ, seg_pos={0 0}, upload=439163]: Long tail cancellation due to optimal threshold Dec 26 11:51:49.230812872 (500h22m49.101895907s) [noise=true , address=45.91.116.9:4262 , piece_id=67TKAHQYFTJSF5WIOW6VAPOCQ66ECKQLL34OXSKBZHLWY4FKR4VA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 11:51:49.230819655 (500h22m49.1019027s) [noise=true , address=45.91.116.9:4262 , piece_id=67TKAHQYFTJSF5WIOW6VAPOCQ66ECKQLL34OXSKBZHLWY4FKR4VA, seg_pos={0 0}, upload=439163]: Long tail cancellation due to optimal threshold Dec 26 11:51:49.230836246 (500h22m49.101919291s) [seg_pos={0 0}, upload=439163]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:51:49.230905047 (500h22m49.101988092s) [upload=439163]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:51:49.230909446 (500h22m49.101992481s) [upload=439163]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:51:49.530474351 (500h22m49.401557396s) [upload=439163]: Done uploading segments... Dec 26 11:51:49.530495973 (500h22m49.401579018s) [upload=439163]: Upload finished. err: <nil> Duration: 5.865923388s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................................X....X............] duration: 4.450285702s trace: https://jaeger.dev.storj.io/trace/10a24935a8f16015 FAIL (2025-12-26 11:54:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.450309157s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................................X....X...........] duration: 3.834464748s trace: https://jaeger.dev.storj.io/trace/10a24935a8f16015 FAIL (2025-12-26 11:53:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.834487491s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 1.39442227s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 11:53:36 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.394451106s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.XX......XXX.XX......X.X..XX.X.X...X....X......X.X.X...X.X..] duration: 5.9819639s trace: https://jaeger.dev.storj.io/trace/28e3ccf01bc9e478 FAIL (2025-12-26 11:53:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: IZRHL6SKMFALQ75VLSR6ZBGNSQKSH6F4RZB7KXW5RWI5G7KNR5YA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: WKISHW5AVXBJPXICXH3LVTFAB4DO72INFTRWP67JTGLL4HBSPQUA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 7L2XKXGUBTS763VQUC5AYRR27YTJOW4BJTYVOHN24H54JZC7LKZQ) context deadline exceeded ======LOGS====== Dec 26 11:51:45.806103832 (500h22m45.677186887s) [upload=439163]: Starting upload Dec 26 11:51:45.806222699 (500h22m45.677305754s) [upload=439163]: Done starting upload Dec 26 11:51:45.806269387 (500h22m45.677352463s) [upload=439163]: Uploading segments... Dec 26 11:51:45.806469509 (500h22m45.677552554s) [upload=439163]: Got next segment. Inline: false Dec 26 11:51:45.806510096 (500h22m45.677593141s) [upload=439163]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:51:45.806516167 (500h22m45.677599212s) [upload=439163]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:51:46.053749784 (500h22m45.92483283s) [seg_pos={0 0}, upload=439163]: Begin upload segment... Dec 26 11:51:46.053825158 (500h22m45.924908203s) [seg_pos={0 0}, upload=439163]: Done begin upload segment. Dec 26 11:51:46.053853322 (500h22m45.924936357s) [noise=true , address=145.79.227.33:4870 , piece_id=GJUKOF3IYXKWYTAJDONYXYR5H2QFOWSMJ5PFP2F6QBQ4ZMB74ZGQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.053898177 (500h22m45.924981242s) [noise=true , address=45.91.116.4:4937 , piece_id=N2LTAF2SR7LSDVAGLK2KURMN5BSK6HURPDEXL4WXGZMZXAR5DAMQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.054859197 (500h22m45.925942242s) [noise=true , address=145.79.227.31:4799 , piece_id=KLIQEQJ67N3NTFHF4GNAW7MEUEJ22YZOWNCIZASR7ADELUA4UJ5Q, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.055169739 (500h22m45.926252784s) [noise=true , address=145.79.227.12:4765 , piece_id=DFUA3GX2F5WFZRHWWKRWDWNLK4KYVEEZMMNLYMOLKMUNOWNVGZMQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.055444532 (500h22m45.926527567s) [noise=true , address=145.79.227.15:4875 , piece_id=FGC37BWAWWLNS6FX2RDPCDJ2FCMGY7E5JMCB7QNRT3Q2ENIBQ5EA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.055695850 (500h22m45.926778895s) [noise=true , address=145.79.227.11:4732 , piece_id=SBNRQTRG2YRJH7Y7SD3UX5TI3AFLUCVCXBLJL65PVSAHKIM2RLAQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.056146729 (500h22m45.927229764s) [noise=true , address=145.79.227.49:4337 , piece_id=NCNSJ5VWGAMZ4IPZCSK7T4PCGKC4IJHUNIRSQGHXUXH433KDD5KQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.056454064 (500h22m45.927537099s) [noise=true , address=145.79.227.52:4446 , piece_id=CQ5ZYB5QYRENMRV4MP76KYIJECIU2HTW7IO2QWOULX77DYETYRRA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.057571462 (500h22m45.928654507s) [noise=true , address=45.91.116.11:4118 , piece_id=WKISHW5AVXBJPXICXH3LVTFAB4DO72INFTRWP67JTGLL4HBSPQUA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.058400641 (500h22m45.929483686s) [noise=true , address=145.79.227.51:4410 , piece_id=BQLHMZT7ZRBOFPVDUMY3IL42IPJQ6E3FSDWYSSI3HHB6J6MUISYQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.058824157 (500h22m45.929907202s) [noise=true , address=145.79.227.46:4603 , piece_id=MD3G53QYNDJVYMJ6ERS5E5GM5R7V55XEAWKMYJGXKCVUG7KADGZQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.058936591 (500h22m45.930019626s) [noise=true , address=45.91.116.9:4262 , piece_id=67TKAHQYFTJSF5WIOW6VAPOCQ66ECKQLL34OXSKBZHLWY4FKR4VA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.059430712 (500h22m45.930513747s) [noise=true , address=145.79.227.17:4970 , piece_id=2Q77VM35EUEUDHR47FGAPCBBHEX3N3QFGOGY5GF6PNEFP6ZMCU2Q, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.059797260 (500h22m45.930880295s) [noise=true , address=77.90.31.13:4627 , piece_id=7L2XKXGUBTS763VQUC5AYRR27YTJOW4BJTYVOHN24H54JZC7LKZQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.060238070 (500h22m45.931321105s) [noise=true , address=77.90.31.10:4519 , piece_id=IZRHL6SKMFALQ75VLSR6ZBGNSQKSH6F4RZB7KXW5RWI5G7KNR5YA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.060510879 (500h22m45.931593924s) [noise=true , address=145.79.227.44:4193 , piece_id=JXRRRGDODZYWSDUJM4WX6RTZ2TWABCFDYFO6RKBWRFICXLAG6TVQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:47.674430101 (500h22m47.545513146s) [upload=439163]: Next returned nil segment Dec 26 11:51:47.674444337 (500h22m47.545527372s) [upload=439163]: Waiting for error group managing segments... Dec 26 11:51:48.058568168 (500h22m47.929651203s) [noise=true , address=145.79.227.46:4603 , piece_id=MD3G53QYNDJVYMJ6ERS5E5GM5R7V55XEAWKMYJGXKCVUG7KADGZQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.058901202 (500h22m47.929984237s) [noise=true , address=145.79.227.12:4765 , piece_id=DFUA3GX2F5WFZRHWWKRWDWNLK4KYVEEZMMNLYMOLKMUNOWNVGZMQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.060248368 (500h22m47.931331413s) [noise=true , address=145.79.227.15:4875 , piece_id=FGC37BWAWWLNS6FX2RDPCDJ2FCMGY7E5JMCB7QNRT3Q2ENIBQ5EA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.060436035 (500h22m47.93151908s) [noise=true , address=145.79.227.11:4732 , piece_id=SBNRQTRG2YRJH7Y7SD3UX5TI3AFLUCVCXBLJL65PVSAHKIM2RLAQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.063895081 (500h22m47.934978126s) [noise=true , address=145.79.227.49:4337 , piece_id=NCNSJ5VWGAMZ4IPZCSK7T4PCGKC4IJHUNIRSQGHXUXH433KDD5KQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.067448307 (500h22m47.938531352s) [noise=true , address=145.79.227.17:4970 , piece_id=2Q77VM35EUEUDHR47FGAPCBBHEX3N3QFGOGY5GF6PNEFP6ZMCU2Q, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.067687342 (500h22m47.938770377s) [noise=true , address=145.79.227.31:4799 , piece_id=KLIQEQJ67N3NTFHF4GNAW7MEUEJ22YZOWNCIZASR7ADELUA4UJ5Q, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.069843568 (500h22m47.940926603s) [noise=true , address=145.79.227.52:4446 , piece_id=CQ5ZYB5QYRENMRV4MP76KYIJECIU2HTW7IO2QWOULX77DYETYRRA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.079236143 (500h22m47.950319189s) [noise=true , address=145.79.227.51:4410 , piece_id=BQLHMZT7ZRBOFPVDUMY3IL42IPJQ6E3FSDWYSSI3HHB6J6MUISYQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.273171081 (500h22m48.144254126s) [noise=true , address=77.90.31.10:4519 , piece_id=IZRHL6SKMFALQ75VLSR6ZBGNSQKSH6F4RZB7KXW5RWI5G7KNR5YA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.337012426 (500h22m48.208095471s) [noise=true , address=45.91.116.11:4118 , piece_id=WKISHW5AVXBJPXICXH3LVTFAB4DO72INFTRWP67JTGLL4HBSPQUA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.655203850 (500h22m48.526286895s) [noise=true , address=45.91.116.4:4937 , piece_id=N2LTAF2SR7LSDVAGLK2KURMN5BSK6HURPDEXL4WXGZMZXAR5DAMQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:49.230200346 (500h22m49.101283381s) [noise=true , address=77.90.31.13:4627 , piece_id=7L2XKXGUBTS763VQUC5AYRR27YTJOW4BJTYVOHN24H54JZC7LKZQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:49.230227107 (500h22m49.101310152s) [seg_pos={0 0}, upload=439163]: Segment reached optimal threshold of 13 pieces. Dec 26 11:51:49.230425084 (500h22m49.101508129s) [noise=true , address=145.79.227.33:4870 , piece_id=GJUKOF3IYXKWYTAJDONYXYR5H2QFOWSMJ5PFP2F6QBQ4ZMB74ZGQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 11:51:49.230432528 (500h22m49.101515563s) [noise=true , address=145.79.227.33:4870 , piece_id=GJUKOF3IYXKWYTAJDONYXYR5H2QFOWSMJ5PFP2F6QBQ4ZMB74ZGQ, seg_pos={0 0}, upload=439163]: Long tail cancellation due to optimal threshold Dec 26 11:51:49.230576362 (500h22m49.101659407s) [noise=true , address=145.79.227.44:4193 , piece_id=JXRRRGDODZYWSDUJM4WX6RTZ2TWABCFDYFO6RKBWRFICXLAG6TVQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 11:51:49.230583695 (500h22m49.101666731s) [noise=true , address=145.79.227.44:4193 , piece_id=JXRRRGDODZYWSDUJM4WX6RTZ2TWABCFDYFO6RKBWRFICXLAG6TVQ, seg_pos={0 0}, upload=439163]: Long tail cancellation due to optimal threshold Dec 26 11:51:49.230812872 (500h22m49.101895907s) [noise=true , address=45.91.116.9:4262 , piece_id=67TKAHQYFTJSF5WIOW6VAPOCQ66ECKQLL34OXSKBZHLWY4FKR4VA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 11:51:49.230819655 (500h22m49.1019027s) [noise=true , address=45.91.116.9:4262 , piece_id=67TKAHQYFTJSF5WIOW6VAPOCQ66ECKQLL34OXSKBZHLWY4FKR4VA, seg_pos={0 0}, upload=439163]: Long tail cancellation due to optimal threshold Dec 26 11:51:49.230836246 (500h22m49.101919291s) [seg_pos={0 0}, upload=439163]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:51:49.230905047 (500h22m49.101988092s) [upload=439163]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:51:49.230909446 (500h22m49.101992481s) [upload=439163]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:51:49.530474351 (500h22m49.401557396s) [upload=439163]: Done uploading segments... Dec 26 11:51:49.530495973 (500h22m49.401579018s) [upload=439163]: Upload finished. err: <nil> Duration: 5.981995499s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................................X....X..........] duration: 4.519226107s trace: https://jaeger.dev.storj.io/trace/10a24935a8f16015 FAIL (2025-12-26 11:52:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.519246917s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..XX......XXX.XX......X.X..XX.X.X...X....X......X.X.X...X.X.] duration: 5.74486387s trace: https://jaeger.dev.storj.io/trace/28e3ccf01bc9e478 FAIL (2025-12-26 11:52:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: IZRHL6SKMFALQ75VLSR6ZBGNSQKSH6F4RZB7KXW5RWI5G7KNR5YA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: WKISHW5AVXBJPXICXH3LVTFAB4DO72INFTRWP67JTGLL4HBSPQUA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 7L2XKXGUBTS763VQUC5AYRR27YTJOW4BJTYVOHN24H54JZC7LKZQ) context deadline exceeded ======LOGS====== Dec 26 11:51:45.806103832 (500h22m45.677186887s) [upload=439163]: Starting upload Dec 26 11:51:45.806222699 (500h22m45.677305754s) [upload=439163]: Done starting upload Dec 26 11:51:45.806269387 (500h22m45.677352463s) [upload=439163]: Uploading segments... Dec 26 11:51:45.806469509 (500h22m45.677552554s) [upload=439163]: Got next segment. Inline: false Dec 26 11:51:45.806510096 (500h22m45.677593141s) [upload=439163]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:51:45.806516167 (500h22m45.677599212s) [upload=439163]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:51:46.053749784 (500h22m45.92483283s) [seg_pos={0 0}, upload=439163]: Begin upload segment... Dec 26 11:51:46.053825158 (500h22m45.924908203s) [seg_pos={0 0}, upload=439163]: Done begin upload segment. Dec 26 11:51:46.053853322 (500h22m45.924936357s) [noise=true , address=145.79.227.33:4870 , piece_id=GJUKOF3IYXKWYTAJDONYXYR5H2QFOWSMJ5PFP2F6QBQ4ZMB74ZGQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.053898177 (500h22m45.924981242s) [noise=true , address=45.91.116.4:4937 , piece_id=N2LTAF2SR7LSDVAGLK2KURMN5BSK6HURPDEXL4WXGZMZXAR5DAMQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.054859197 (500h22m45.925942242s) [noise=true , address=145.79.227.31:4799 , piece_id=KLIQEQJ67N3NTFHF4GNAW7MEUEJ22YZOWNCIZASR7ADELUA4UJ5Q, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.055169739 (500h22m45.926252784s) [noise=true , address=145.79.227.12:4765 , piece_id=DFUA3GX2F5WFZRHWWKRWDWNLK4KYVEEZMMNLYMOLKMUNOWNVGZMQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.055444532 (500h22m45.926527567s) [noise=true , address=145.79.227.15:4875 , piece_id=FGC37BWAWWLNS6FX2RDPCDJ2FCMGY7E5JMCB7QNRT3Q2ENIBQ5EA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.055695850 (500h22m45.926778895s) [noise=true , address=145.79.227.11:4732 , piece_id=SBNRQTRG2YRJH7Y7SD3UX5TI3AFLUCVCXBLJL65PVSAHKIM2RLAQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.056146729 (500h22m45.927229764s) [noise=true , address=145.79.227.49:4337 , piece_id=NCNSJ5VWGAMZ4IPZCSK7T4PCGKC4IJHUNIRSQGHXUXH433KDD5KQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.056454064 (500h22m45.927537099s) [noise=true , address=145.79.227.52:4446 , piece_id=CQ5ZYB5QYRENMRV4MP76KYIJECIU2HTW7IO2QWOULX77DYETYRRA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.057571462 (500h22m45.928654507s) [noise=true , address=45.91.116.11:4118 , piece_id=WKISHW5AVXBJPXICXH3LVTFAB4DO72INFTRWP67JTGLL4HBSPQUA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.058400641 (500h22m45.929483686s) [noise=true , address=145.79.227.51:4410 , piece_id=BQLHMZT7ZRBOFPVDUMY3IL42IPJQ6E3FSDWYSSI3HHB6J6MUISYQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.058824157 (500h22m45.929907202s) [noise=true , address=145.79.227.46:4603 , piece_id=MD3G53QYNDJVYMJ6ERS5E5GM5R7V55XEAWKMYJGXKCVUG7KADGZQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.058936591 (500h22m45.930019626s) [noise=true , address=45.91.116.9:4262 , piece_id=67TKAHQYFTJSF5WIOW6VAPOCQ66ECKQLL34OXSKBZHLWY4FKR4VA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.059430712 (500h22m45.930513747s) [noise=true , address=145.79.227.17:4970 , piece_id=2Q77VM35EUEUDHR47FGAPCBBHEX3N3QFGOGY5GF6PNEFP6ZMCU2Q, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.059797260 (500h22m45.930880295s) [noise=true , address=77.90.31.13:4627 , piece_id=7L2XKXGUBTS763VQUC5AYRR27YTJOW4BJTYVOHN24H54JZC7LKZQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.060238070 (500h22m45.931321105s) [noise=true , address=77.90.31.10:4519 , piece_id=IZRHL6SKMFALQ75VLSR6ZBGNSQKSH6F4RZB7KXW5RWI5G7KNR5YA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.060510879 (500h22m45.931593924s) [noise=true , address=145.79.227.44:4193 , piece_id=JXRRRGDODZYWSDUJM4WX6RTZ2TWABCFDYFO6RKBWRFICXLAG6TVQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:47.674430101 (500h22m47.545513146s) [upload=439163]: Next returned nil segment Dec 26 11:51:47.674444337 (500h22m47.545527372s) [upload=439163]: Waiting for error group managing segments... Dec 26 11:51:48.058568168 (500h22m47.929651203s) [noise=true , address=145.79.227.46:4603 , piece_id=MD3G53QYNDJVYMJ6ERS5E5GM5R7V55XEAWKMYJGXKCVUG7KADGZQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.058901202 (500h22m47.929984237s) [noise=true , address=145.79.227.12:4765 , piece_id=DFUA3GX2F5WFZRHWWKRWDWNLK4KYVEEZMMNLYMOLKMUNOWNVGZMQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.060248368 (500h22m47.931331413s) [noise=true , address=145.79.227.15:4875 , piece_id=FGC37BWAWWLNS6FX2RDPCDJ2FCMGY7E5JMCB7QNRT3Q2ENIBQ5EA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.060436035 (500h22m47.93151908s) [noise=true , address=145.79.227.11:4732 , piece_id=SBNRQTRG2YRJH7Y7SD3UX5TI3AFLUCVCXBLJL65PVSAHKIM2RLAQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.063895081 (500h22m47.934978126s) [noise=true , address=145.79.227.49:4337 , piece_id=NCNSJ5VWGAMZ4IPZCSK7T4PCGKC4IJHUNIRSQGHXUXH433KDD5KQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.067448307 (500h22m47.938531352s) [noise=true , address=145.79.227.17:4970 , piece_id=2Q77VM35EUEUDHR47FGAPCBBHEX3N3QFGOGY5GF6PNEFP6ZMCU2Q, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.067687342 (500h22m47.938770377s) [noise=true , address=145.79.227.31:4799 , piece_id=KLIQEQJ67N3NTFHF4GNAW7MEUEJ22YZOWNCIZASR7ADELUA4UJ5Q, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.069843568 (500h22m47.940926603s) [noise=true , address=145.79.227.52:4446 , piece_id=CQ5ZYB5QYRENMRV4MP76KYIJECIU2HTW7IO2QWOULX77DYETYRRA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.079236143 (500h22m47.950319189s) [noise=true , address=145.79.227.51:4410 , piece_id=BQLHMZT7ZRBOFPVDUMY3IL42IPJQ6E3FSDWYSSI3HHB6J6MUISYQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.273171081 (500h22m48.144254126s) [noise=true , address=77.90.31.10:4519 , piece_id=IZRHL6SKMFALQ75VLSR6ZBGNSQKSH6F4RZB7KXW5RWI5G7KNR5YA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.337012426 (500h22m48.208095471s) [noise=true , address=45.91.116.11:4118 , piece_id=WKISHW5AVXBJPXICXH3LVTFAB4DO72INFTRWP67JTGLL4HBSPQUA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.655203850 (500h22m48.526286895s) [noise=true , address=45.91.116.4:4937 , piece_id=N2LTAF2SR7LSDVAGLK2KURMN5BSK6HURPDEXL4WXGZMZXAR5DAMQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:49.230200346 (500h22m49.101283381s) [noise=true , address=77.90.31.13:4627 , piece_id=7L2XKXGUBTS763VQUC5AYRR27YTJOW4BJTYVOHN24H54JZC7LKZQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:49.230227107 (500h22m49.101310152s) [seg_pos={0 0}, upload=439163]: Segment reached optimal threshold of 13 pieces. Dec 26 11:51:49.230425084 (500h22m49.101508129s) [noise=true , address=145.79.227.33:4870 , piece_id=GJUKOF3IYXKWYTAJDONYXYR5H2QFOWSMJ5PFP2F6QBQ4ZMB74ZGQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 11:51:49.230432528 (500h22m49.101515563s) [noise=true , address=145.79.227.33:4870 , piece_id=GJUKOF3IYXKWYTAJDONYXYR5H2QFOWSMJ5PFP2F6QBQ4ZMB74ZGQ, seg_pos={0 0}, upload=439163]: Long tail cancellation due to optimal threshold Dec 26 11:51:49.230576362 (500h22m49.101659407s) [noise=true , address=145.79.227.44:4193 , piece_id=JXRRRGDODZYWSDUJM4WX6RTZ2TWABCFDYFO6RKBWRFICXLAG6TVQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 11:51:49.230583695 (500h22m49.101666731s) [noise=true , address=145.79.227.44:4193 , piece_id=JXRRRGDODZYWSDUJM4WX6RTZ2TWABCFDYFO6RKBWRFICXLAG6TVQ, seg_pos={0 0}, upload=439163]: Long tail cancellation due to optimal threshold Dec 26 11:51:49.230812872 (500h22m49.101895907s) [noise=true , address=45.91.116.9:4262 , piece_id=67TKAHQYFTJSF5WIOW6VAPOCQ66ECKQLL34OXSKBZHLWY4FKR4VA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 11:51:49.230819655 (500h22m49.1019027s) [noise=true , address=45.91.116.9:4262 , piece_id=67TKAHQYFTJSF5WIOW6VAPOCQ66ECKQLL34OXSKBZHLWY4FKR4VA, seg_pos={0 0}, upload=439163]: Long tail cancellation due to optimal threshold Dec 26 11:51:49.230836246 (500h22m49.101919291s) [seg_pos={0 0}, upload=439163]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:51:49.230905047 (500h22m49.101988092s) [upload=439163]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:51:49.230909446 (500h22m49.101992481s) [upload=439163]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:51:49.530474351 (500h22m49.401557396s) [upload=439163]: Done uploading segments... Dec 26 11:51:49.530495973 (500h22m49.401579018s) [upload=439163]: Upload finished. err: <nil> Duration: 5.74489059s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 1.401475101s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 11:52:39 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.401507412s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...XX......XXX.XX......X.X..XX.X.X...X....X......X.X.X...X.X] duration: 30.002834306s trace: https://jaeger.dev.storj.io/trace/28e3ccf01bc9e478 FAIL (2025-12-26 11:52:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: IZRHL6SKMFALQ75VLSR6ZBGNSQKSH6F4RZB7KXW5RWI5G7KNR5YA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: WKISHW5AVXBJPXICXH3LVTFAB4DO72INFTRWP67JTGLL4HBSPQUA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 7L2XKXGUBTS763VQUC5AYRR27YTJOW4BJTYVOHN24H54JZC7LKZQ) context deadline exceeded ======LOGS====== Dec 26 11:51:45.806103832 (500h22m45.677186887s) [upload=439163]: Starting upload Dec 26 11:51:45.806222699 (500h22m45.677305754s) [upload=439163]: Done starting upload Dec 26 11:51:45.806269387 (500h22m45.677352463s) [upload=439163]: Uploading segments... Dec 26 11:51:45.806469509 (500h22m45.677552554s) [upload=439163]: Got next segment. Inline: false Dec 26 11:51:45.806510096 (500h22m45.677593141s) [upload=439163]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:51:45.806516167 (500h22m45.677599212s) [upload=439163]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:51:46.053749784 (500h22m45.92483283s) [seg_pos={0 0}, upload=439163]: Begin upload segment... Dec 26 11:51:46.053825158 (500h22m45.924908203s) [seg_pos={0 0}, upload=439163]: Done begin upload segment. Dec 26 11:51:46.053853322 (500h22m45.924936357s) [noise=true , address=145.79.227.33:4870 , piece_id=GJUKOF3IYXKWYTAJDONYXYR5H2QFOWSMJ5PFP2F6QBQ4ZMB74ZGQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.053898177 (500h22m45.924981242s) [noise=true , address=45.91.116.4:4937 , piece_id=N2LTAF2SR7LSDVAGLK2KURMN5BSK6HURPDEXL4WXGZMZXAR5DAMQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.054859197 (500h22m45.925942242s) [noise=true , address=145.79.227.31:4799 , piece_id=KLIQEQJ67N3NTFHF4GNAW7MEUEJ22YZOWNCIZASR7ADELUA4UJ5Q, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.055169739 (500h22m45.926252784s) [noise=true , address=145.79.227.12:4765 , piece_id=DFUA3GX2F5WFZRHWWKRWDWNLK4KYVEEZMMNLYMOLKMUNOWNVGZMQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.055444532 (500h22m45.926527567s) [noise=true , address=145.79.227.15:4875 , piece_id=FGC37BWAWWLNS6FX2RDPCDJ2FCMGY7E5JMCB7QNRT3Q2ENIBQ5EA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.055695850 (500h22m45.926778895s) [noise=true , address=145.79.227.11:4732 , piece_id=SBNRQTRG2YRJH7Y7SD3UX5TI3AFLUCVCXBLJL65PVSAHKIM2RLAQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.056146729 (500h22m45.927229764s) [noise=true , address=145.79.227.49:4337 , piece_id=NCNSJ5VWGAMZ4IPZCSK7T4PCGKC4IJHUNIRSQGHXUXH433KDD5KQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.056454064 (500h22m45.927537099s) [noise=true , address=145.79.227.52:4446 , piece_id=CQ5ZYB5QYRENMRV4MP76KYIJECIU2HTW7IO2QWOULX77DYETYRRA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.057571462 (500h22m45.928654507s) [noise=true , address=45.91.116.11:4118 , piece_id=WKISHW5AVXBJPXICXH3LVTFAB4DO72INFTRWP67JTGLL4HBSPQUA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.058400641 (500h22m45.929483686s) [noise=true , address=145.79.227.51:4410 , piece_id=BQLHMZT7ZRBOFPVDUMY3IL42IPJQ6E3FSDWYSSI3HHB6J6MUISYQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.058824157 (500h22m45.929907202s) [noise=true , address=145.79.227.46:4603 , piece_id=MD3G53QYNDJVYMJ6ERS5E5GM5R7V55XEAWKMYJGXKCVUG7KADGZQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.058936591 (500h22m45.930019626s) [noise=true , address=45.91.116.9:4262 , piece_id=67TKAHQYFTJSF5WIOW6VAPOCQ66ECKQLL34OXSKBZHLWY4FKR4VA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.059430712 (500h22m45.930513747s) [noise=true , address=145.79.227.17:4970 , piece_id=2Q77VM35EUEUDHR47FGAPCBBHEX3N3QFGOGY5GF6PNEFP6ZMCU2Q, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.059797260 (500h22m45.930880295s) [noise=true , address=77.90.31.13:4627 , piece_id=7L2XKXGUBTS763VQUC5AYRR27YTJOW4BJTYVOHN24H54JZC7LKZQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.060238070 (500h22m45.931321105s) [noise=true , address=77.90.31.10:4519 , piece_id=IZRHL6SKMFALQ75VLSR6ZBGNSQKSH6F4RZB7KXW5RWI5G7KNR5YA, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:46.060510879 (500h22m45.931593924s) [noise=true , address=145.79.227.44:4193 , piece_id=JXRRRGDODZYWSDUJM4WX6RTZ2TWABCFDYFO6RKBWRFICXLAG6TVQ, seg_pos={0 0}, upload=439163]: Uploading piece... Dec 26 11:51:47.674430101 (500h22m47.545513146s) [upload=439163]: Next returned nil segment Dec 26 11:51:47.674444337 (500h22m47.545527372s) [upload=439163]: Waiting for error group managing segments... Dec 26 11:51:48.058568168 (500h22m47.929651203s) [noise=true , address=145.79.227.46:4603 , piece_id=MD3G53QYNDJVYMJ6ERS5E5GM5R7V55XEAWKMYJGXKCVUG7KADGZQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.058901202 (500h22m47.929984237s) [noise=true , address=145.79.227.12:4765 , piece_id=DFUA3GX2F5WFZRHWWKRWDWNLK4KYVEEZMMNLYMOLKMUNOWNVGZMQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.060248368 (500h22m47.931331413s) [noise=true , address=145.79.227.15:4875 , piece_id=FGC37BWAWWLNS6FX2RDPCDJ2FCMGY7E5JMCB7QNRT3Q2ENIBQ5EA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.060436035 (500h22m47.93151908s) [noise=true , address=145.79.227.11:4732 , piece_id=SBNRQTRG2YRJH7Y7SD3UX5TI3AFLUCVCXBLJL65PVSAHKIM2RLAQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.063895081 (500h22m47.934978126s) [noise=true , address=145.79.227.49:4337 , piece_id=NCNSJ5VWGAMZ4IPZCSK7T4PCGKC4IJHUNIRSQGHXUXH433KDD5KQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.067448307 (500h22m47.938531352s) [noise=true , address=145.79.227.17:4970 , piece_id=2Q77VM35EUEUDHR47FGAPCBBHEX3N3QFGOGY5GF6PNEFP6ZMCU2Q, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.067687342 (500h22m47.938770377s) [noise=true , address=145.79.227.31:4799 , piece_id=KLIQEQJ67N3NTFHF4GNAW7MEUEJ22YZOWNCIZASR7ADELUA4UJ5Q, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.069843568 (500h22m47.940926603s) [noise=true , address=145.79.227.52:4446 , piece_id=CQ5ZYB5QYRENMRV4MP76KYIJECIU2HTW7IO2QWOULX77DYETYRRA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.079236143 (500h22m47.950319189s) [noise=true , address=145.79.227.51:4410 , piece_id=BQLHMZT7ZRBOFPVDUMY3IL42IPJQ6E3FSDWYSSI3HHB6J6MUISYQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.273171081 (500h22m48.144254126s) [noise=true , address=77.90.31.10:4519 , piece_id=IZRHL6SKMFALQ75VLSR6ZBGNSQKSH6F4RZB7KXW5RWI5G7KNR5YA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.337012426 (500h22m48.208095471s) [noise=true , address=45.91.116.11:4118 , piece_id=WKISHW5AVXBJPXICXH3LVTFAB4DO72INFTRWP67JTGLL4HBSPQUA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:48.655203850 (500h22m48.526286895s) [noise=true , address=45.91.116.4:4937 , piece_id=N2LTAF2SR7LSDVAGLK2KURMN5BSK6HURPDEXL4WXGZMZXAR5DAMQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:49.230200346 (500h22m49.101283381s) [noise=true , address=77.90.31.13:4627 , piece_id=7L2XKXGUBTS763VQUC5AYRR27YTJOW4BJTYVOHN24H54JZC7LKZQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: <nil> Dec 26 11:51:49.230227107 (500h22m49.101310152s) [seg_pos={0 0}, upload=439163]: Segment reached optimal threshold of 13 pieces. Dec 26 11:51:49.230425084 (500h22m49.101508129s) [noise=true , address=145.79.227.33:4870 , piece_id=GJUKOF3IYXKWYTAJDONYXYR5H2QFOWSMJ5PFP2F6QBQ4ZMB74ZGQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 11:51:49.230432528 (500h22m49.101515563s) [noise=true , address=145.79.227.33:4870 , piece_id=GJUKOF3IYXKWYTAJDONYXYR5H2QFOWSMJ5PFP2F6QBQ4ZMB74ZGQ, seg_pos={0 0}, upload=439163]: Long tail cancellation due to optimal threshold Dec 26 11:51:49.230576362 (500h22m49.101659407s) [noise=true , address=145.79.227.44:4193 , piece_id=JXRRRGDODZYWSDUJM4WX6RTZ2TWABCFDYFO6RKBWRFICXLAG6TVQ, seg_pos={0 0}, upload=439163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 11:51:49.230583695 (500h22m49.101666731s) [noise=true , address=145.79.227.44:4193 , piece_id=JXRRRGDODZYWSDUJM4WX6RTZ2TWABCFDYFO6RKBWRFICXLAG6TVQ, seg_pos={0 0}, upload=439163]: Long tail cancellation due to optimal threshold Dec 26 11:51:49.230812872 (500h22m49.101895907s) [noise=true , address=45.91.116.9:4262 , piece_id=67TKAHQYFTJSF5WIOW6VAPOCQ66ECKQLL34OXSKBZHLWY4FKR4VA, seg_pos={0 0}, upload=439163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 11:51:49.230819655 (500h22m49.1019027s) [noise=true , address=45.91.116.9:4262 , piece_id=67TKAHQYFTJSF5WIOW6VAPOCQ66ECKQLL34OXSKBZHLWY4FKR4VA, seg_pos={0 0}, upload=439163]: Long tail cancellation due to optimal threshold Dec 26 11:51:49.230836246 (500h22m49.101919291s) [seg_pos={0 0}, upload=439163]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:51:49.230905047 (500h22m49.101988092s) [upload=439163]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:51:49.230909446 (500h22m49.101992481s) [upload=439163]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:51:49.530474351 (500h22m49.401557396s) [upload=439163]: Done uploading segments... Dec 26 11:51:49.530495973 (500h22m49.401579018s) [upload=439163]: Upload finished. err: <nil> Duration: 30.002854694s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................................X....X.........] duration: 4.06334676s trace: https://jaeger.dev.storj.io/trace/10a24935a8f16015 FAIL (2025-12-26 11:51:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.06337267s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 1.390755857s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 11:51:28 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.390786347s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................................X....X........] duration: 3.505807175s trace: https://jaeger.dev.storj.io/trace/10a24935a8f16015 FAIL (2025-12-26 11:50:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.50594127s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [....XX......XXX.XX......X.X..XX.X.X...X....X......X.X.X...X.] duration: 5.765674718s trace: https://jaeger.dev.storj.io/trace/7f0af36264f92ae3 FAIL (2025-12-26 11:50:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 11:50:02.657073920 (500h21m2.528156965s) [upload=439138]: Starting upload Dec 26 11:50:02.657155545 (500h21m2.5282386s) [upload=439138]: Done starting upload Dec 26 11:50:02.657264513 (500h21m2.528347558s) [upload=439138]: Uploading segments... Dec 26 11:50:02.657560577 (500h21m2.528643622s) [upload=439138]: Got next segment. Inline: false Dec 26 11:50:02.657645178 (500h21m2.528728223s) [upload=439138]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:50:02.657654134 (500h21m2.528737179s) [upload=439138]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:50:02.904283081 (500h21m2.775366126s) [seg_pos={0 0}, upload=439138]: Begin upload segment... Dec 26 11:50:02.904346552 (500h21m2.775429597s) [seg_pos={0 0}, upload=439138]: Done begin upload segment. Dec 26 11:50:02.904383161 (500h21m2.775466206s) [noise=true , address=45.91.116.5:4299 , piece_id=6X2ZS4A7D72UPFZAZE2S6456INBJKLO2G37UVJNIRDZGJACEBIMA, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.906970919 (500h21m2.778053964s) [noise=true , address=145.79.227.49:4338 , piece_id=AMHWI4WQ2JW7XADY2TR6POWZQS6KXY5I7UQCA3KTDLOQ7EATIFGQ, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.907893305 (500h21m2.778976351s) [noise=true , address=145.79.227.15:4874 , piece_id=4LRKDAYHMS6KN4ISP65MQIALMWVD4T7RFQOWTMZXT53MOS3ETS6A, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.908421020 (500h21m2.779504065s) [noise=true , address=45.91.116.12:4225 , piece_id=Z7XOQ7SROTYDP4TVZA2GWN54RVAB5DCBGDYE3U3AD7SVPIOIMQ7Q, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.909373755 (500h21m2.78045681s) [noise=true , address=145.79.227.41:4086 , piece_id=NFARQ7OXANEXO6VFNYZGYHBYZ7KJMRKBMFLVH25QOHVMYRFKMABA, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.910112051 (500h21m2.781195096s) [noise=true , address=45.91.116.9:4264 , piece_id=G6MMJJMYLMHO33ARNYHZ56BG5GAHN6R753C767R4UNYP44SHDONQ, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.911590475 (500h21m2.78267353s) [noise=true , address=77.90.31.19:4844 , piece_id=JF3XZSQMGMDVQUQYUDJGU56RCTRDF2N327S4N7PIDTUB72NTTM7A, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.912224082 (500h21m2.783307127s) [noise=true , address=145.79.227.37:4567 , piece_id=CC2SQXRJ3SGAWFLORD6C73NLPLOYO3QDWXD3BYBJ54O67J4Y333Q, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.913320540 (500h21m2.784403585s) [noise=true , address=45.91.116.14:4189 , piece_id=YX2S5HCV3FA2YNF74UOL55WEZAZBERGSO23VUBF5BKRUYTL7PZYQ, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.914005585 (500h21m2.78508863s) [noise=true , address=145.79.227.4:4187 , piece_id=ZGTODJP5SHCBYAGMIR74LBIZGOV2P23GEYGWBGZOYOAXCSSEMUTQ, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.915491223 (500h21m2.786574268s) [noise=true , address=145.79.227.51:4409 , piece_id=KFSA4SZKTZE4CGANF4MF5XKDL3FX6ABT4DQSPRWFPRBEKITVCFAQ, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.915596984 (500h21m2.786680029s) [noise=true , address=45.91.116.13:4155 , piece_id=MWOPMFEKZBM66PCSAJLTZCDD57JUZLHC3L2J5EKSGIC4YTDFK2DA, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.917174138 (500h21m2.788257183s) [noise=true , address=45.91.116.6:4976 , piece_id=BOONEGCQKISDPQ73AT6BTJA75NOS76CACWWO62UNHDQFQDNKV34A, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.917700350 (500h21m2.788783405s) [noise=true , address=145.79.227.13:4803 , piece_id=W5ZMPDUPBVHZXVYNUX7MGGJJU3DBRI6TNXDZO7OTB32P2HSYESMA, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.918968665 (500h21m2.79005171s) [noise=true , address=45.91.116.7:4009 , piece_id=SPAENHBLMDZBJRKBSN5LO6IJPKOCZCQTR7ZGUDR7YICIPBQ7F5IA, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.920590553 (500h21m2.791673599s) [noise=true , address=145.79.227.16:4909 , piece_id=7DHGOCBW3VADIIMSCXPVJMRBDXATJWSY56E6YGGBI7KRU6DE7WNA, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:04.546869130 (500h21m4.417952175s) [upload=439138]: Next returned nil segment Dec 26 11:50:04.546887524 (500h21m4.417970569s) [upload=439138]: Waiting for error group managing segments... Dec 26 11:50:04.947743276 (500h21m4.818826331s) [noise=true , address=145.79.227.4:4187 , piece_id=ZGTODJP5SHCBYAGMIR74LBIZGOV2P23GEYGWBGZOYOAXCSSEMUTQ, seg_pos={0 0}, upload=439138]: Done uploading piece. err: <nil> Dec 26 11:50:04.948210336 (500h21m4.819293381s) [noise=true , address=145.79.227.41:4086 , piece_id=NFARQ7OXANEXO6VFNYZGYHBYZ7KJMRKBMFLVH25QOHVMYRFKMABA, seg_pos={0 0}, upload=439138]: Done uploading piece. err: <nil> Dec 26 11:50:04.948532849 (500h21m4.819615894s) [noise=true , address=145.79.227.16:4909 , piece_id=7DHGOCBW3VADIIMSCXPVJMRBDXATJWSY56E6YGGBI7KRU6DE7WNA, seg_pos={0 0}, upload=439138]: Done uploading piece. err: <nil> Dec 26 11:50:04.949244193 (500h21m4.820327239s) [noise=true , address=145.79.227.13:4803 , piece_id=W5ZMPDUPBVHZXVYNUX7MGGJJU3DBRI6TNXDZO7OTB32P2HSYESMA, seg_pos={0 0}, upload=439138]: Done uploading piece. err: <nil> Dec 26 11:50:04.955472844 (500h21m4.826555889s) [noise=true , address=145.79.227.37:4567 , piece_id=CC2SQXRJ3SGAWFLORD6C73NLPLOYO3QDWXD3BYBJ54O67J4Y333Q, seg_pos={0 0}, upload=439138]: Done uploading piece. err: <nil> Dec 26 11:50:04.955856384 (500h21m4.826939429s) [noise=true , address=145.79.227.49:4338 , piece_id=AMHWI4WQ2JW7XADY2TR6POWZQS6KXY5I7UQCA3KTDLOQ7EATIFGQ, seg_pos={0 0}, upload=439138]: Done uploading piece. err: <nil> Dec 26 11:50:04.978822887 (500h21m4.849905932s) [noise=true , address=145.79.227.15:4874 , piece_id=4LRKDAYHMS6KN4ISP65MQIALMWVD4T7RFQOWTMZXT53MOS3ETS6A, seg_pos={0 0}, upload=439138]: Done uploading piece. err: <nil> Dec 26 11:50:04.985451358 (500h21m4.856534393s) [noise=true , address=145.79.227.51:4409 , piece_id=KFSA4SZKTZE4CGANF4MF5XKDL3FX6ABT4DQSPRWFPRBEKITVCFAQ, seg_pos={0 0}, upload=439138]: Done uploading piece. err: <nil> Dec 26 11:50:05.132732940 (500h21m5.003815995s) [noise=true , address=77.90.31.19:4844 , piece_id=JF3XZSQMGMDVQUQYUDJGU56RCTRDF2N327S4N7PIDTUB72NTTM7A, seg_pos={0 0}, upload=439138]: Done uploading piece. err: <nil> Dec 26 11:50:05.169890404 (500h21m5.040973449s) [noise=true , address=45.91.116.9:4264 , piece_id=G6MMJJMYLMHO33ARNYHZ56BG5GAHN6R753C767R4UNYP44SHDONQ, seg_pos={0 0}, upload=439138]: Done uploading piece. err: <nil> Dec 26 11:50:05.197969030 (500h21m5.069052076s) [noise=true , address=45.91.116.14:4189 , piece_id=YX2S5HCV3FA2YNF74UOL55WEZAZBERGSO23VUBF5BKRUYTL7PZYQ, seg_pos={0 0}, upload=439138]: Done uploading piece. err: <nil> Dec 26 11:50:05.204507661 (500h21m5.075590706s) [noise=true , address=45.91.116.13:4155 , piece_id=MWOPMFEKZBM66PCSAJLTZCDD57JUZLHC3L2J5EKSGIC4YTDFK2DA, seg_pos={0 0}, upload=439138]: Done uploading piece. err: <nil> Dec 26 11:50:05.246981689 (500h21m5.118064744s) [noise=true , address=45.91.116.5:4299 , piece_id=6X2ZS4A7D72UPFZAZE2S6456INBJKLO2G37UVJNIRDZGJACEBIMA, seg_pos={0 0}, upload=439138]: Done uploading piece. err: <nil> Dec 26 11:50:05.247007789 (500h21m5.118090834s) [seg_pos={0 0}, upload=439138]: Segment reached optimal threshold of 13 pieces. Dec 26 11:50:05.247277663 (500h21m5.118360718s) [noise=true , address=45.91.116.6:4976 , piece_id=BOONEGCQKISDPQ73AT6BTJA75NOS76CACWWO62UNHDQFQDNKV34A, seg_pos={0 0}, upload=439138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 26 11:50:05.247286790 (500h21m5.118369835s) [noise=true , address=45.91.116.6:4976 , piece_id=BOONEGCQKISDPQ73AT6BTJA75NOS76CACWWO62UNHDQFQDNKV34A, seg_pos={0 0}, upload=439138]: Long tail cancellation due to optimal threshold Dec 26 11:50:05.247592002 (500h21m5.118675057s) [noise=true , address=45.91.116.7:4009 , piece_id=SPAENHBLMDZBJRKBSN5LO6IJPKOCZCQTR7ZGUDR7YICIPBQ7F5IA, seg_pos={0 0}, upload=439138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:38176->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:38176->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38176->45.91.116.7:4009: use of closed network connection Dec 26 11:50:05.247601980 (500h21m5.118685035s) [noise=true , address=45.91.116.7:4009 , piece_id=SPAENHBLMDZBJRKBSN5LO6IJPKOCZCQTR7ZGUDR7YICIPBQ7F5IA, seg_pos={0 0}, upload=439138]: Long tail cancellation due to optimal threshold Dec 26 11:50:05.247690690 (500h21m5.118773745s) [noise=true , address=45.91.116.12:4225 , piece_id=Z7XOQ7SROTYDP4TVZA2GWN54RVAB5DCBGDYE3U3AD7SVPIOIMQ7Q, seg_pos={0 0}, upload=439138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 11:50:05.247731036 (500h21m5.118814081s) [noise=true , address=45.91.116.12:4225 , piece_id=Z7XOQ7SROTYDP4TVZA2GWN54RVAB5DCBGDYE3U3AD7SVPIOIMQ7Q, seg_pos={0 0}, upload=439138]: Long tail cancellation due to optimal threshold Dec 26 11:50:05.247765702 (500h21m5.118848748s) [seg_pos={0 0}, upload=439138]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:50:05.247952959 (500h21m5.119036004s) [upload=439138]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:50:05.247962406 (500h21m5.119045451s) [upload=439138]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:50:05.543779235 (500h21m5.41486228s) [upload=439138]: Done uploading segments... Dec 26 11:50:05.543832076 (500h21m5.414915121s) [upload=439138]: Upload finished. err: <nil> Duration: 5.765708552s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.....XX......XXX.XX......X.X..XX.X.X...X....X......X.X.X...X] duration: 30.003974765s trace: https://jaeger.dev.storj.io/trace/7f0af36264f92ae3 FAIL (2025-12-26 11:50:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 11:50:02.657073920 (500h21m2.528156965s) [upload=439138]: Starting upload Dec 26 11:50:02.657155545 (500h21m2.5282386s) [upload=439138]: Done starting upload Dec 26 11:50:02.657264513 (500h21m2.528347558s) [upload=439138]: Uploading segments... Dec 26 11:50:02.657560577 (500h21m2.528643622s) [upload=439138]: Got next segment. Inline: false Dec 26 11:50:02.657645178 (500h21m2.528728223s) [upload=439138]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:50:02.657654134 (500h21m2.528737179s) [upload=439138]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:50:02.904283081 (500h21m2.775366126s) [seg_pos={0 0}, upload=439138]: Begin upload segment... Dec 26 11:50:02.904346552 (500h21m2.775429597s) [seg_pos={0 0}, upload=439138]: Done begin upload segment. Dec 26 11:50:02.904383161 (500h21m2.775466206s) [noise=true , address=45.91.116.5:4299 , piece_id=6X2ZS4A7D72UPFZAZE2S6456INBJKLO2G37UVJNIRDZGJACEBIMA, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.906970919 (500h21m2.778053964s) [noise=true , address=145.79.227.49:4338 , piece_id=AMHWI4WQ2JW7XADY2TR6POWZQS6KXY5I7UQCA3KTDLOQ7EATIFGQ, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.907893305 (500h21m2.778976351s) [noise=true , address=145.79.227.15:4874 , piece_id=4LRKDAYHMS6KN4ISP65MQIALMWVD4T7RFQOWTMZXT53MOS3ETS6A, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.908421020 (500h21m2.779504065s) [noise=true , address=45.91.116.12:4225 , piece_id=Z7XOQ7SROTYDP4TVZA2GWN54RVAB5DCBGDYE3U3AD7SVPIOIMQ7Q, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.909373755 (500h21m2.78045681s) [noise=true , address=145.79.227.41:4086 , piece_id=NFARQ7OXANEXO6VFNYZGYHBYZ7KJMRKBMFLVH25QOHVMYRFKMABA, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.910112051 (500h21m2.781195096s) [noise=true , address=45.91.116.9:4264 , piece_id=G6MMJJMYLMHO33ARNYHZ56BG5GAHN6R753C767R4UNYP44SHDONQ, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.911590475 (500h21m2.78267353s) [noise=true , address=77.90.31.19:4844 , piece_id=JF3XZSQMGMDVQUQYUDJGU56RCTRDF2N327S4N7PIDTUB72NTTM7A, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.912224082 (500h21m2.783307127s) [noise=true , address=145.79.227.37:4567 , piece_id=CC2SQXRJ3SGAWFLORD6C73NLPLOYO3QDWXD3BYBJ54O67J4Y333Q, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.913320540 (500h21m2.784403585s) [noise=true , address=45.91.116.14:4189 , piece_id=YX2S5HCV3FA2YNF74UOL55WEZAZBERGSO23VUBF5BKRUYTL7PZYQ, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.914005585 (500h21m2.78508863s) [noise=true , address=145.79.227.4:4187 , piece_id=ZGTODJP5SHCBYAGMIR74LBIZGOV2P23GEYGWBGZOYOAXCSSEMUTQ, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.915491223 (500h21m2.786574268s) [noise=true , address=145.79.227.51:4409 , piece_id=KFSA4SZKTZE4CGANF4MF5XKDL3FX6ABT4DQSPRWFPRBEKITVCFAQ, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.915596984 (500h21m2.786680029s) [noise=true , address=45.91.116.13:4155 , piece_id=MWOPMFEKZBM66PCSAJLTZCDD57JUZLHC3L2J5EKSGIC4YTDFK2DA, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.917174138 (500h21m2.788257183s) [noise=true , address=45.91.116.6:4976 , piece_id=BOONEGCQKISDPQ73AT6BTJA75NOS76CACWWO62UNHDQFQDNKV34A, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.917700350 (500h21m2.788783405s) [noise=true , address=145.79.227.13:4803 , piece_id=W5ZMPDUPBVHZXVYNUX7MGGJJU3DBRI6TNXDZO7OTB32P2HSYESMA, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.918968665 (500h21m2.79005171s) [noise=true , address=45.91.116.7:4009 , piece_id=SPAENHBLMDZBJRKBSN5LO6IJPKOCZCQTR7ZGUDR7YICIPBQ7F5IA, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:02.920590553 (500h21m2.791673599s) [noise=true , address=145.79.227.16:4909 , piece_id=7DHGOCBW3VADIIMSCXPVJMRBDXATJWSY56E6YGGBI7KRU6DE7WNA, seg_pos={0 0}, upload=439138]: Uploading piece... Dec 26 11:50:04.546869130 (500h21m4.417952175s) [upload=439138]: Next returned nil segment Dec 26 11:50:04.546887524 (500h21m4.417970569s) [upload=439138]: Waiting for error group managing segments... Dec 26 11:50:04.947743276 (500h21m4.818826331s) [noise=true , address=145.79.227.4:4187 , piece_id=ZGTODJP5SHCBYAGMIR74LBIZGOV2P23GEYGWBGZOYOAXCSSEMUTQ, seg_pos={0 0}, upload=439138]: Done uploading piece. err: <nil> Dec 26 11:50:04.948210336 (500h21m4.819293381s) [noise=true , address=145.79.227.41:4086 , piece_id=NFARQ7OXANEXO6VFNYZGYHBYZ7KJMRKBMFLVH25QOHVMYRFKMABA, seg_pos={0 0}, upload=439138]: Done uploading piece. err: <nil> Dec 26 11:50:04.948532849 (500h21m4.819615894s) [noise=true , address=145.79.227.16:4909 , piece_id=7DHGOCBW3VADIIMSCXPVJMRBDXATJWSY56E6YGGBI7KRU6DE7WNA, seg_pos={0 0}, upload=439138]: Done uploading piece. err: <nil> Dec 26 11:50:04.949244193 (500h21m4.820327239s) [noise=true , address=145.79.227.13:4803 , piece_id=W5ZMPDUPBVHZXVYNUX7MGGJJU3DBRI6TNXDZO7OTB32P2HSYESMA, seg_pos={0 0}, upload=439138]: Done uploading piece. err: <nil> Dec 26 11:50:04.955472844 (500h21m4.826555889s) [noise=true , address=145.79.227.37:4567 , piece_id=CC2SQXRJ3SGAWFLORD6C73NLPLOYO3QDWXD3BYBJ54O67J4Y333Q, seg_pos={0 0}, upload=439138]: Done uploading piece. err: <nil> Dec 26 11:50:04.955856384 (500h21m4.826939429s) [noise=true , address=145.79.227.49:4338 , piece_id=AMHWI4WQ2JW7XADY2TR6POWZQS6KXY5I7UQCA3KTDLOQ7EATIFGQ, seg_pos={0 0}, upload=439138]: Done uploading piece. err: <nil> Dec 26 11:50:04.978822887 (500h21m4.849905932s) [noise=true , address=145.79.227.15:4874 , piece_id=4LRKDAYHMS6KN4ISP65MQIALMWVD4T7RFQOWTMZXT53MOS3ETS6A, seg_pos={0 0}, upload=439138]: Done uploading piece. err: <nil> Dec 26 11:50:04.985451358 (500h21m4.856534393s) [noise=true , address=145.79.227.51:4409 , piece_id=KFSA4SZKTZE4CGANF4MF5XKDL3FX6ABT4DQSPRWFPRBEKITVCFAQ, seg_pos={0 0}, upload=439138]: Done uploading piece. err: <nil> Dec 26 11:50:05.132732940 (500h21m5.003815995s) [noise=true , address=77.90.31.19:4844 , piece_id=JF3XZSQMGMDVQUQYUDJGU56RCTRDF2N327S4N7PIDTUB72NTTM7A, seg_pos={0 0}, upload=439138]: Done uploading piece. err: <nil> Dec 26 11:50:05.169890404 (500h21m5.040973449s) [noise=true , address=45.91.116.9:4264 , piece_id=G6MMJJMYLMHO33ARNYHZ56BG5GAHN6R753C767R4UNYP44SHDONQ, seg_pos={0 0}, upload=439138]: Done uploading piece. err: <nil> Dec 26 11:50:05.197969030 (500h21m5.069052076s) [noise=true , address=45.91.116.14:4189 , piece_id=YX2S5HCV3FA2YNF74UOL55WEZAZBERGSO23VUBF5BKRUYTL7PZYQ, seg_pos={0 0}, upload=439138]: Done uploading piece. err: <nil> Dec 26 11:50:05.204507661 (500h21m5.075590706s) [noise=true , address=45.91.116.13:4155 , piece_id=MWOPMFEKZBM66PCSAJLTZCDD57JUZLHC3L2J5EKSGIC4YTDFK2DA, seg_pos={0 0}, upload=439138]: Done uploading piece. err: <nil> Dec 26 11:50:05.246981689 (500h21m5.118064744s) [noise=true , address=45.91.116.5:4299 , piece_id=6X2ZS4A7D72UPFZAZE2S6456INBJKLO2G37UVJNIRDZGJACEBIMA, seg_pos={0 0}, upload=439138]: Done uploading piece. err: <nil> Dec 26 11:50:05.247007789 (500h21m5.118090834s) [seg_pos={0 0}, upload=439138]: Segment reached optimal threshold of 13 pieces. Dec 26 11:50:05.247277663 (500h21m5.118360718s) [noise=true , address=45.91.116.6:4976 , piece_id=BOONEGCQKISDPQ73AT6BTJA75NOS76CACWWO62UNHDQFQDNKV34A, seg_pos={0 0}, upload=439138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 26 11:50:05.247286790 (500h21m5.118369835s) [noise=true , address=45.91.116.6:4976 , piece_id=BOONEGCQKISDPQ73AT6BTJA75NOS76CACWWO62UNHDQFQDNKV34A, seg_pos={0 0}, upload=439138]: Long tail cancellation due to optimal threshold Dec 26 11:50:05.247592002 (500h21m5.118675057s) [noise=true , address=45.91.116.7:4009 , piece_id=SPAENHBLMDZBJRKBSN5LO6IJPKOCZCQTR7ZGUDR7YICIPBQ7F5IA, seg_pos={0 0}, upload=439138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:38176->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:38176->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38176->45.91.116.7:4009: use of closed network connection Dec 26 11:50:05.247601980 (500h21m5.118685035s) [noise=true , address=45.91.116.7:4009 , piece_id=SPAENHBLMDZBJRKBSN5LO6IJPKOCZCQTR7ZGUDR7YICIPBQ7F5IA, seg_pos={0 0}, upload=439138]: Long tail cancellation due to optimal threshold Dec 26 11:50:05.247690690 (500h21m5.118773745s) [noise=true , address=45.91.116.12:4225 , piece_id=Z7XOQ7SROTYDP4TVZA2GWN54RVAB5DCBGDYE3U3AD7SVPIOIMQ7Q, seg_pos={0 0}, upload=439138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 11:50:05.247731036 (500h21m5.118814081s) [noise=true , address=45.91.116.12:4225 , piece_id=Z7XOQ7SROTYDP4TVZA2GWN54RVAB5DCBGDYE3U3AD7SVPIOIMQ7Q, seg_pos={0 0}, upload=439138]: Long tail cancellation due to optimal threshold Dec 26 11:50:05.247765702 (500h21m5.118848748s) [seg_pos={0 0}, upload=439138]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:50:05.247952959 (500h21m5.119036004s) [upload=439138]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:50:05.247962406 (500h21m5.119045451s) [upload=439138]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:50:05.543779235 (500h21m5.41486228s) [upload=439138]: Done uploading segments... Dec 26 11:50:05.543832076 (500h21m5.414915121s) [upload=439138]: Upload finished. err: <nil> Duration: 30.004008881s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 1.401436737s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 11:50:29 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.40145479s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 1.398148865s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 11:49:59 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.398172672s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................................X....X.......] duration: 4.080783895s trace: https://jaeger.dev.storj.io/trace/10a24935a8f16015 FAIL (2025-12-26 11:49:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.080810877s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.422614282s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 11:49:39 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.422641334s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [......XX......XXX.XX......X.X..XX.X.X...X....X......X.X.X...] duration: 6.070165743s trace: https://jaeger.dev.storj.io/trace/35234c041cadbaf FAIL (2025-12-26 11:49:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 11:46:56.696152839 (500h17m56.567235884s) [upload=439093]: Starting upload Dec 26 11:46:56.696237520 (500h17m56.567320555s) [upload=439093]: Done starting upload Dec 26 11:46:56.696299999 (500h17m56.567383044s) [upload=439093]: Uploading segments... Dec 26 11:46:56.696549154 (500h17m56.567632199s) [upload=439093]: Got next segment. Inline: false Dec 26 11:46:56.696589971 (500h17m56.567673016s) [upload=439093]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:46:56.696596023 (500h17m56.567679068s) [upload=439093]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:46:56.944668216 (500h17m56.815751251s) [seg_pos={0 0}, upload=439093]: Begin upload segment... Dec 26 11:46:56.944719344 (500h17m56.815802379s) [seg_pos={0 0}, upload=439093]: Done begin upload segment. Dec 26 11:46:56.944739571 (500h17m56.815822617s) [noise=true , address=45.91.116.13:4156 , piece_id=QQX3ZH372HRMWSBJUZAFHI2AZOS43RQ7PXHIPWQXHSRYE45US4NA, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.944750413 (500h17m56.815833448s) [noise=true , address=145.79.227.12:4765 , piece_id=3QY2FW65JIAC6GJFJWSQ2MWB6IBPUJXUUVUEJOICFCPG6DZUYMLQ, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.944860813 (500h17m56.815943868s) [noise=true , address=45.91.116.14:4192 , piece_id=4F2T25NTGIJZ7JSXO2T6J5SFQ6FJEH3B4I65QOVFBXE46OHVVA7A, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.945426259 (500h17m56.816509314s) [noise=true , address=145.79.227.45:4231 , piece_id=QY3EL4UBOXHUA3TQZ3NIIWK7MKYHTWJGGE6MHLTLKVX5NIXWU7HQ, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.945675203 (500h17m56.816758268s) [noise=true , address=145.79.227.40:4050 , piece_id=SG7WZQGR3BAKWXBJYMTETTPSL3G4CYLR7JGZS6YLWCW7T6SCMKIA, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.945985935 (500h17m56.81706898s) [noise=true , address=145.79.227.34:4906 , piece_id=PKPLPFOT7IH55544YJFZCJJBQCFB5GXEYLNJB5B43ECU46XPMAUQ, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.946218518 (500h17m56.817301563s) [noise=true , address=145.79.227.17:4971 , piece_id=5JLBY4JSKGBGEUG2PYUDBVX2YL5U6SFXMCCAWYSA2ZGTO2VGFDYQ, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.946344918 (500h17m56.817427953s) [noise=true , address=145.79.227.16:4912 , piece_id=Y7DGGMUOOKBN4KM32QANMLZLDF2ETZ5MYROR5IVJOSAEFUBUAIYA, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.946835313 (500h17m56.817918348s) [noise=true , address=145.79.227.50:4373 , piece_id=7JXLSVS5QLQG6YHTWZ4VRTQ2ATO2JSSNXPT4HIH5MYYJ7PQM6CDA, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.946845582 (500h17m56.817928627s) [noise=true , address=45.91.116.5:4300 , piece_id=TVBREMHSNQHOSXBN3XL6OHRHX3HODOETCWZ2BLZZIFVVSZ4257RQ, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.947250723 (500h17m56.818333768s) [noise=true , address=145.79.227.39:4014 , piece_id=C6H2TJK5HYGJNL3Q7LYZTM2MSKP7FBZJEPGGUOZWXQB42WADBREA, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.947660504 (500h17m56.818743549s) [noise=true , address=145.79.227.20:4439 , piece_id=PRCIGDS7YT7AY6L25LZ4CBQ2PVD4AZMIXMGQ5XXUJZZQ4VTIDP5A, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.948006422 (500h17m56.819089467s) [noise=true , address=45.91.116.4:4940 , piece_id=37LJ66K2WGWCXNXWBJBQOYVUGOKR3XUINJIJWTMJEBCNFBHSFSIA, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.948402587 (500h17m56.819485622s) [noise=true , address=145.79.227.2:4113 , piece_id=TDCOQJ4HLGHZPTAFRN76ZHUS7ZVNBTNO2ZD2CJE6XTJKXZY6FJIQ, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.948793071 (500h17m56.819876106s) [noise=true , address=77.90.31.15:4700 , piece_id=KNTRZ433WA4RGQMPV4TOTXVI4NYEYHHWENJCEZU4ZUU74NFDX3KQ, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.949174797 (500h17m56.820257833s) [noise=true , address=145.79.227.51:4411 , piece_id=5H3ZQU5JGUJKUEIPWLEHT7Y2Z6CJBJSPDISJNL77F3ZQVNDOUAUA, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:58.659906146 (500h17m58.530989191s) [upload=439093]: Next returned nil segment Dec 26 11:46:58.659924540 (500h17m58.531007585s) [upload=439093]: Waiting for error group managing segments... Dec 26 11:46:58.943963151 (500h17m58.815046186s) [noise=true , address=145.79.227.17:4971 , piece_id=5JLBY4JSKGBGEUG2PYUDBVX2YL5U6SFXMCCAWYSA2ZGTO2VGFDYQ, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.948494690 (500h17m58.819577725s) [noise=true , address=145.79.227.51:4411 , piece_id=5H3ZQU5JGUJKUEIPWLEHT7Y2Z6CJBJSPDISJNL77F3ZQVNDOUAUA, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.951401525 (500h17m58.82248456s) [noise=true , address=145.79.227.12:4765 , piece_id=3QY2FW65JIAC6GJFJWSQ2MWB6IBPUJXUUVUEJOICFCPG6DZUYMLQ, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.956946003 (500h17m58.828029038s) [noise=true , address=145.79.227.34:4906 , piece_id=PKPLPFOT7IH55544YJFZCJJBQCFB5GXEYLNJB5B43ECU46XPMAUQ, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.957240343 (500h17m58.828323398s) [noise=true , address=145.79.227.20:4439 , piece_id=PRCIGDS7YT7AY6L25LZ4CBQ2PVD4AZMIXMGQ5XXUJZZQ4VTIDP5A, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.957902243 (500h17m58.828985288s) [noise=true , address=145.79.227.2:4113 , piece_id=TDCOQJ4HLGHZPTAFRN76ZHUS7ZVNBTNO2ZD2CJE6XTJKXZY6FJIQ, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.958055986 (500h17m58.829139031s) [noise=true , address=145.79.227.45:4231 , piece_id=QY3EL4UBOXHUA3TQZ3NIIWK7MKYHTWJGGE6MHLTLKVX5NIXWU7HQ, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.959094714 (500h17m58.83017777s) [noise=true , address=145.79.227.39:4014 , piece_id=C6H2TJK5HYGJNL3Q7LYZTM2MSKP7FBZJEPGGUOZWXQB42WADBREA, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.959788295 (500h17m58.83087136s) [noise=true , address=145.79.227.16:4912 , piece_id=Y7DGGMUOOKBN4KM32QANMLZLDF2ETZ5MYROR5IVJOSAEFUBUAIYA, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.963958746 (500h17m58.835041791s) [noise=true , address=145.79.227.50:4373 , piece_id=7JXLSVS5QLQG6YHTWZ4VRTQ2ATO2JSSNXPT4HIH5MYYJ7PQM6CDA, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:59.176989905 (500h17m59.04807295s) [noise=true , address=45.91.116.14:4192 , piece_id=4F2T25NTGIJZ7JSXO2T6J5SFQ6FJEH3B4I65QOVFBXE46OHVVA7A, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:59.517256223 (500h17m59.388339268s) [noise=true , address=77.90.31.15:4700 , piece_id=KNTRZ433WA4RGQMPV4TOTXVI4NYEYHHWENJCEZU4ZUU74NFDX3KQ, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:59.846070014 (500h17m59.717153049s) [noise=true , address=45.91.116.13:4156 , piece_id=QQX3ZH372HRMWSBJUZAFHI2AZOS43RQ7PXHIPWQXHSRYE45US4NA, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:59.846090001 (500h17m59.717173036s) [seg_pos={0 0}, upload=439093]: Segment reached optimal threshold of 13 pieces. Dec 26 11:46:59.846305031 (500h17m59.717388076s) [noise=true , address=45.91.116.5:4300 , piece_id=TVBREMHSNQHOSXBN3XL6OHRHX3HODOETCWZ2BLZZIFVVSZ4257RQ, seg_pos={0 0}, upload=439093]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:55962->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:55962->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55962->45.91.116.5:4300: use of closed network connection Dec 26 11:46:59.846312235 (500h17m59.71739529s) [noise=true , address=45.91.116.5:4300 , piece_id=TVBREMHSNQHOSXBN3XL6OHRHX3HODOETCWZ2BLZZIFVVSZ4257RQ, seg_pos={0 0}, upload=439093]: Long tail cancellation due to optimal threshold Dec 26 11:46:59.846564475 (500h17m59.71764752s) [noise=true , address=145.79.227.40:4050 , piece_id=SG7WZQGR3BAKWXBJYMTETTPSL3G4CYLR7JGZS6YLWCW7T6SCMKIA, seg_pos={0 0}, upload=439093]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 11:46:59.846583922 (500h17m59.717666968s) [noise=true , address=145.79.227.40:4050 , piece_id=SG7WZQGR3BAKWXBJYMTETTPSL3G4CYLR7JGZS6YLWCW7T6SCMKIA, seg_pos={0 0}, upload=439093]: Long tail cancellation due to optimal threshold Dec 26 11:46:59.846927106 (500h17m59.718010151s) [noise=true , address=45.91.116.4:4940 , piece_id=37LJ66K2WGWCXNXWBJBQOYVUGOKR3XUINJIJWTMJEBCNFBHSFSIA, seg_pos={0 0}, upload=439093]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 11:46:59.846937526 (500h17m59.718020571s) [noise=true , address=45.91.116.4:4940 , piece_id=37LJ66K2WGWCXNXWBJBQOYVUGOKR3XUINJIJWTMJEBCNFBHSFSIA, seg_pos={0 0}, upload=439093]: Long tail cancellation due to optimal threshold Dec 26 11:46:59.846963375 (500h17m59.71804642s) [seg_pos={0 0}, upload=439093]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:46:59.847041654 (500h17m59.718124699s) [upload=439093]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:46:59.847048066 (500h17m59.718131101s) [upload=439093]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:47:00.138103137 (500h18m0.009186182s) [upload=439093]: Done uploading segments... Dec 26 11:47:00.138127123 (500h18m0.009210158s) [upload=439093]: Upload finished. err: <nil> Duration: 6.070193096s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................................X....X......] duration: 5.000731989s trace: https://jaeger.dev.storj.io/trace/10a24935a8f16015 FAIL (2025-12-26 11:48:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000750504s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 1.4401481s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 11:48:32 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.440179879s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X......XX......XXX.XX......X.X..XX.X.X...X....X......X.X.X..] duration: 6.044904389s trace: https://jaeger.dev.storj.io/trace/35234c041cadbaf FAIL (2025-12-26 11:48:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 11:46:56.696152839 (500h17m56.567235884s) [upload=439093]: Starting upload Dec 26 11:46:56.696237520 (500h17m56.567320555s) [upload=439093]: Done starting upload Dec 26 11:46:56.696299999 (500h17m56.567383044s) [upload=439093]: Uploading segments... Dec 26 11:46:56.696549154 (500h17m56.567632199s) [upload=439093]: Got next segment. Inline: false Dec 26 11:46:56.696589971 (500h17m56.567673016s) [upload=439093]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:46:56.696596023 (500h17m56.567679068s) [upload=439093]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:46:56.944668216 (500h17m56.815751251s) [seg_pos={0 0}, upload=439093]: Begin upload segment... Dec 26 11:46:56.944719344 (500h17m56.815802379s) [seg_pos={0 0}, upload=439093]: Done begin upload segment. Dec 26 11:46:56.944739571 (500h17m56.815822617s) [noise=true , address=45.91.116.13:4156 , piece_id=QQX3ZH372HRMWSBJUZAFHI2AZOS43RQ7PXHIPWQXHSRYE45US4NA, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.944750413 (500h17m56.815833448s) [noise=true , address=145.79.227.12:4765 , piece_id=3QY2FW65JIAC6GJFJWSQ2MWB6IBPUJXUUVUEJOICFCPG6DZUYMLQ, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.944860813 (500h17m56.815943868s) [noise=true , address=45.91.116.14:4192 , piece_id=4F2T25NTGIJZ7JSXO2T6J5SFQ6FJEH3B4I65QOVFBXE46OHVVA7A, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.945426259 (500h17m56.816509314s) [noise=true , address=145.79.227.45:4231 , piece_id=QY3EL4UBOXHUA3TQZ3NIIWK7MKYHTWJGGE6MHLTLKVX5NIXWU7HQ, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.945675203 (500h17m56.816758268s) [noise=true , address=145.79.227.40:4050 , piece_id=SG7WZQGR3BAKWXBJYMTETTPSL3G4CYLR7JGZS6YLWCW7T6SCMKIA, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.945985935 (500h17m56.81706898s) [noise=true , address=145.79.227.34:4906 , piece_id=PKPLPFOT7IH55544YJFZCJJBQCFB5GXEYLNJB5B43ECU46XPMAUQ, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.946218518 (500h17m56.817301563s) [noise=true , address=145.79.227.17:4971 , piece_id=5JLBY4JSKGBGEUG2PYUDBVX2YL5U6SFXMCCAWYSA2ZGTO2VGFDYQ, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.946344918 (500h17m56.817427953s) [noise=true , address=145.79.227.16:4912 , piece_id=Y7DGGMUOOKBN4KM32QANMLZLDF2ETZ5MYROR5IVJOSAEFUBUAIYA, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.946835313 (500h17m56.817918348s) [noise=true , address=145.79.227.50:4373 , piece_id=7JXLSVS5QLQG6YHTWZ4VRTQ2ATO2JSSNXPT4HIH5MYYJ7PQM6CDA, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.946845582 (500h17m56.817928627s) [noise=true , address=45.91.116.5:4300 , piece_id=TVBREMHSNQHOSXBN3XL6OHRHX3HODOETCWZ2BLZZIFVVSZ4257RQ, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.947250723 (500h17m56.818333768s) [noise=true , address=145.79.227.39:4014 , piece_id=C6H2TJK5HYGJNL3Q7LYZTM2MSKP7FBZJEPGGUOZWXQB42WADBREA, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.947660504 (500h17m56.818743549s) [noise=true , address=145.79.227.20:4439 , piece_id=PRCIGDS7YT7AY6L25LZ4CBQ2PVD4AZMIXMGQ5XXUJZZQ4VTIDP5A, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.948006422 (500h17m56.819089467s) [noise=true , address=45.91.116.4:4940 , piece_id=37LJ66K2WGWCXNXWBJBQOYVUGOKR3XUINJIJWTMJEBCNFBHSFSIA, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.948402587 (500h17m56.819485622s) [noise=true , address=145.79.227.2:4113 , piece_id=TDCOQJ4HLGHZPTAFRN76ZHUS7ZVNBTNO2ZD2CJE6XTJKXZY6FJIQ, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.948793071 (500h17m56.819876106s) [noise=true , address=77.90.31.15:4700 , piece_id=KNTRZ433WA4RGQMPV4TOTXVI4NYEYHHWENJCEZU4ZUU74NFDX3KQ, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.949174797 (500h17m56.820257833s) [noise=true , address=145.79.227.51:4411 , piece_id=5H3ZQU5JGUJKUEIPWLEHT7Y2Z6CJBJSPDISJNL77F3ZQVNDOUAUA, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:58.659906146 (500h17m58.530989191s) [upload=439093]: Next returned nil segment Dec 26 11:46:58.659924540 (500h17m58.531007585s) [upload=439093]: Waiting for error group managing segments... Dec 26 11:46:58.943963151 (500h17m58.815046186s) [noise=true , address=145.79.227.17:4971 , piece_id=5JLBY4JSKGBGEUG2PYUDBVX2YL5U6SFXMCCAWYSA2ZGTO2VGFDYQ, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.948494690 (500h17m58.819577725s) [noise=true , address=145.79.227.51:4411 , piece_id=5H3ZQU5JGUJKUEIPWLEHT7Y2Z6CJBJSPDISJNL77F3ZQVNDOUAUA, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.951401525 (500h17m58.82248456s) [noise=true , address=145.79.227.12:4765 , piece_id=3QY2FW65JIAC6GJFJWSQ2MWB6IBPUJXUUVUEJOICFCPG6DZUYMLQ, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.956946003 (500h17m58.828029038s) [noise=true , address=145.79.227.34:4906 , piece_id=PKPLPFOT7IH55544YJFZCJJBQCFB5GXEYLNJB5B43ECU46XPMAUQ, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.957240343 (500h17m58.828323398s) [noise=true , address=145.79.227.20:4439 , piece_id=PRCIGDS7YT7AY6L25LZ4CBQ2PVD4AZMIXMGQ5XXUJZZQ4VTIDP5A, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.957902243 (500h17m58.828985288s) [noise=true , address=145.79.227.2:4113 , piece_id=TDCOQJ4HLGHZPTAFRN76ZHUS7ZVNBTNO2ZD2CJE6XTJKXZY6FJIQ, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.958055986 (500h17m58.829139031s) [noise=true , address=145.79.227.45:4231 , piece_id=QY3EL4UBOXHUA3TQZ3NIIWK7MKYHTWJGGE6MHLTLKVX5NIXWU7HQ, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.959094714 (500h17m58.83017777s) [noise=true , address=145.79.227.39:4014 , piece_id=C6H2TJK5HYGJNL3Q7LYZTM2MSKP7FBZJEPGGUOZWXQB42WADBREA, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.959788295 (500h17m58.83087136s) [noise=true , address=145.79.227.16:4912 , piece_id=Y7DGGMUOOKBN4KM32QANMLZLDF2ETZ5MYROR5IVJOSAEFUBUAIYA, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.963958746 (500h17m58.835041791s) [noise=true , address=145.79.227.50:4373 , piece_id=7JXLSVS5QLQG6YHTWZ4VRTQ2ATO2JSSNXPT4HIH5MYYJ7PQM6CDA, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:59.176989905 (500h17m59.04807295s) [noise=true , address=45.91.116.14:4192 , piece_id=4F2T25NTGIJZ7JSXO2T6J5SFQ6FJEH3B4I65QOVFBXE46OHVVA7A, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:59.517256223 (500h17m59.388339268s) [noise=true , address=77.90.31.15:4700 , piece_id=KNTRZ433WA4RGQMPV4TOTXVI4NYEYHHWENJCEZU4ZUU74NFDX3KQ, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:59.846070014 (500h17m59.717153049s) [noise=true , address=45.91.116.13:4156 , piece_id=QQX3ZH372HRMWSBJUZAFHI2AZOS43RQ7PXHIPWQXHSRYE45US4NA, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:59.846090001 (500h17m59.717173036s) [seg_pos={0 0}, upload=439093]: Segment reached optimal threshold of 13 pieces. Dec 26 11:46:59.846305031 (500h17m59.717388076s) [noise=true , address=45.91.116.5:4300 , piece_id=TVBREMHSNQHOSXBN3XL6OHRHX3HODOETCWZ2BLZZIFVVSZ4257RQ, seg_pos={0 0}, upload=439093]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:55962->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:55962->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55962->45.91.116.5:4300: use of closed network connection Dec 26 11:46:59.846312235 (500h17m59.71739529s) [noise=true , address=45.91.116.5:4300 , piece_id=TVBREMHSNQHOSXBN3XL6OHRHX3HODOETCWZ2BLZZIFVVSZ4257RQ, seg_pos={0 0}, upload=439093]: Long tail cancellation due to optimal threshold Dec 26 11:46:59.846564475 (500h17m59.71764752s) [noise=true , address=145.79.227.40:4050 , piece_id=SG7WZQGR3BAKWXBJYMTETTPSL3G4CYLR7JGZS6YLWCW7T6SCMKIA, seg_pos={0 0}, upload=439093]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 11:46:59.846583922 (500h17m59.717666968s) [noise=true , address=145.79.227.40:4050 , piece_id=SG7WZQGR3BAKWXBJYMTETTPSL3G4CYLR7JGZS6YLWCW7T6SCMKIA, seg_pos={0 0}, upload=439093]: Long tail cancellation due to optimal threshold Dec 26 11:46:59.846927106 (500h17m59.718010151s) [noise=true , address=45.91.116.4:4940 , piece_id=37LJ66K2WGWCXNXWBJBQOYVUGOKR3XUINJIJWTMJEBCNFBHSFSIA, seg_pos={0 0}, upload=439093]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 11:46:59.846937526 (500h17m59.718020571s) [noise=true , address=45.91.116.4:4940 , piece_id=37LJ66K2WGWCXNXWBJBQOYVUGOKR3XUINJIJWTMJEBCNFBHSFSIA, seg_pos={0 0}, upload=439093]: Long tail cancellation due to optimal threshold Dec 26 11:46:59.846963375 (500h17m59.71804642s) [seg_pos={0 0}, upload=439093]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:46:59.847041654 (500h17m59.718124699s) [upload=439093]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:46:59.847048066 (500h17m59.718131101s) [upload=439093]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:47:00.138103137 (500h18m0.009186182s) [upload=439093]: Done uploading segments... Dec 26 11:47:00.138127123 (500h18m0.009210158s) [upload=439093]: Upload finished. err: <nil> Duration: 6.04493539s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 1.365002699s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 11:48:05 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.365035713s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................................X....X.....] duration: 5.001207866s trace: https://jaeger.dev.storj.io/trace/10a24935a8f16015 FAIL (2025-12-26 11:47:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001296443s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X......XX......XXX.XX......X.X..XX.X.X...X....X......X.X.X.] duration: 6.035649759s trace: https://jaeger.dev.storj.io/trace/35234c041cadbaf FAIL (2025-12-26 11:47:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 11:46:56.696152839 (500h17m56.567235884s) [upload=439093]: Starting upload Dec 26 11:46:56.696237520 (500h17m56.567320555s) [upload=439093]: Done starting upload Dec 26 11:46:56.696299999 (500h17m56.567383044s) [upload=439093]: Uploading segments... Dec 26 11:46:56.696549154 (500h17m56.567632199s) [upload=439093]: Got next segment. Inline: false Dec 26 11:46:56.696589971 (500h17m56.567673016s) [upload=439093]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:46:56.696596023 (500h17m56.567679068s) [upload=439093]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:46:56.944668216 (500h17m56.815751251s) [seg_pos={0 0}, upload=439093]: Begin upload segment... Dec 26 11:46:56.944719344 (500h17m56.815802379s) [seg_pos={0 0}, upload=439093]: Done begin upload segment. Dec 26 11:46:56.944739571 (500h17m56.815822617s) [noise=true , address=45.91.116.13:4156 , piece_id=QQX3ZH372HRMWSBJUZAFHI2AZOS43RQ7PXHIPWQXHSRYE45US4NA, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.944750413 (500h17m56.815833448s) [noise=true , address=145.79.227.12:4765 , piece_id=3QY2FW65JIAC6GJFJWSQ2MWB6IBPUJXUUVUEJOICFCPG6DZUYMLQ, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.944860813 (500h17m56.815943868s) [noise=true , address=45.91.116.14:4192 , piece_id=4F2T25NTGIJZ7JSXO2T6J5SFQ6FJEH3B4I65QOVFBXE46OHVVA7A, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.945426259 (500h17m56.816509314s) [noise=true , address=145.79.227.45:4231 , piece_id=QY3EL4UBOXHUA3TQZ3NIIWK7MKYHTWJGGE6MHLTLKVX5NIXWU7HQ, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.945675203 (500h17m56.816758268s) [noise=true , address=145.79.227.40:4050 , piece_id=SG7WZQGR3BAKWXBJYMTETTPSL3G4CYLR7JGZS6YLWCW7T6SCMKIA, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.945985935 (500h17m56.81706898s) [noise=true , address=145.79.227.34:4906 , piece_id=PKPLPFOT7IH55544YJFZCJJBQCFB5GXEYLNJB5B43ECU46XPMAUQ, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.946218518 (500h17m56.817301563s) [noise=true , address=145.79.227.17:4971 , piece_id=5JLBY4JSKGBGEUG2PYUDBVX2YL5U6SFXMCCAWYSA2ZGTO2VGFDYQ, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.946344918 (500h17m56.817427953s) [noise=true , address=145.79.227.16:4912 , piece_id=Y7DGGMUOOKBN4KM32QANMLZLDF2ETZ5MYROR5IVJOSAEFUBUAIYA, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.946835313 (500h17m56.817918348s) [noise=true , address=145.79.227.50:4373 , piece_id=7JXLSVS5QLQG6YHTWZ4VRTQ2ATO2JSSNXPT4HIH5MYYJ7PQM6CDA, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.946845582 (500h17m56.817928627s) [noise=true , address=45.91.116.5:4300 , piece_id=TVBREMHSNQHOSXBN3XL6OHRHX3HODOETCWZ2BLZZIFVVSZ4257RQ, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.947250723 (500h17m56.818333768s) [noise=true , address=145.79.227.39:4014 , piece_id=C6H2TJK5HYGJNL3Q7LYZTM2MSKP7FBZJEPGGUOZWXQB42WADBREA, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.947660504 (500h17m56.818743549s) [noise=true , address=145.79.227.20:4439 , piece_id=PRCIGDS7YT7AY6L25LZ4CBQ2PVD4AZMIXMGQ5XXUJZZQ4VTIDP5A, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.948006422 (500h17m56.819089467s) [noise=true , address=45.91.116.4:4940 , piece_id=37LJ66K2WGWCXNXWBJBQOYVUGOKR3XUINJIJWTMJEBCNFBHSFSIA, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.948402587 (500h17m56.819485622s) [noise=true , address=145.79.227.2:4113 , piece_id=TDCOQJ4HLGHZPTAFRN76ZHUS7ZVNBTNO2ZD2CJE6XTJKXZY6FJIQ, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.948793071 (500h17m56.819876106s) [noise=true , address=77.90.31.15:4700 , piece_id=KNTRZ433WA4RGQMPV4TOTXVI4NYEYHHWENJCEZU4ZUU74NFDX3KQ, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.949174797 (500h17m56.820257833s) [noise=true , address=145.79.227.51:4411 , piece_id=5H3ZQU5JGUJKUEIPWLEHT7Y2Z6CJBJSPDISJNL77F3ZQVNDOUAUA, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:58.659906146 (500h17m58.530989191s) [upload=439093]: Next returned nil segment Dec 26 11:46:58.659924540 (500h17m58.531007585s) [upload=439093]: Waiting for error group managing segments... Dec 26 11:46:58.943963151 (500h17m58.815046186s) [noise=true , address=145.79.227.17:4971 , piece_id=5JLBY4JSKGBGEUG2PYUDBVX2YL5U6SFXMCCAWYSA2ZGTO2VGFDYQ, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.948494690 (500h17m58.819577725s) [noise=true , address=145.79.227.51:4411 , piece_id=5H3ZQU5JGUJKUEIPWLEHT7Y2Z6CJBJSPDISJNL77F3ZQVNDOUAUA, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.951401525 (500h17m58.82248456s) [noise=true , address=145.79.227.12:4765 , piece_id=3QY2FW65JIAC6GJFJWSQ2MWB6IBPUJXUUVUEJOICFCPG6DZUYMLQ, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.956946003 (500h17m58.828029038s) [noise=true , address=145.79.227.34:4906 , piece_id=PKPLPFOT7IH55544YJFZCJJBQCFB5GXEYLNJB5B43ECU46XPMAUQ, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.957240343 (500h17m58.828323398s) [noise=true , address=145.79.227.20:4439 , piece_id=PRCIGDS7YT7AY6L25LZ4CBQ2PVD4AZMIXMGQ5XXUJZZQ4VTIDP5A, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.957902243 (500h17m58.828985288s) [noise=true , address=145.79.227.2:4113 , piece_id=TDCOQJ4HLGHZPTAFRN76ZHUS7ZVNBTNO2ZD2CJE6XTJKXZY6FJIQ, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.958055986 (500h17m58.829139031s) [noise=true , address=145.79.227.45:4231 , piece_id=QY3EL4UBOXHUA3TQZ3NIIWK7MKYHTWJGGE6MHLTLKVX5NIXWU7HQ, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.959094714 (500h17m58.83017777s) [noise=true , address=145.79.227.39:4014 , piece_id=C6H2TJK5HYGJNL3Q7LYZTM2MSKP7FBZJEPGGUOZWXQB42WADBREA, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.959788295 (500h17m58.83087136s) [noise=true , address=145.79.227.16:4912 , piece_id=Y7DGGMUOOKBN4KM32QANMLZLDF2ETZ5MYROR5IVJOSAEFUBUAIYA, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.963958746 (500h17m58.835041791s) [noise=true , address=145.79.227.50:4373 , piece_id=7JXLSVS5QLQG6YHTWZ4VRTQ2ATO2JSSNXPT4HIH5MYYJ7PQM6CDA, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:59.176989905 (500h17m59.04807295s) [noise=true , address=45.91.116.14:4192 , piece_id=4F2T25NTGIJZ7JSXO2T6J5SFQ6FJEH3B4I65QOVFBXE46OHVVA7A, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:59.517256223 (500h17m59.388339268s) [noise=true , address=77.90.31.15:4700 , piece_id=KNTRZ433WA4RGQMPV4TOTXVI4NYEYHHWENJCEZU4ZUU74NFDX3KQ, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:59.846070014 (500h17m59.717153049s) [noise=true , address=45.91.116.13:4156 , piece_id=QQX3ZH372HRMWSBJUZAFHI2AZOS43RQ7PXHIPWQXHSRYE45US4NA, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:59.846090001 (500h17m59.717173036s) [seg_pos={0 0}, upload=439093]: Segment reached optimal threshold of 13 pieces. Dec 26 11:46:59.846305031 (500h17m59.717388076s) [noise=true , address=45.91.116.5:4300 , piece_id=TVBREMHSNQHOSXBN3XL6OHRHX3HODOETCWZ2BLZZIFVVSZ4257RQ, seg_pos={0 0}, upload=439093]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:55962->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:55962->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55962->45.91.116.5:4300: use of closed network connection Dec 26 11:46:59.846312235 (500h17m59.71739529s) [noise=true , address=45.91.116.5:4300 , piece_id=TVBREMHSNQHOSXBN3XL6OHRHX3HODOETCWZ2BLZZIFVVSZ4257RQ, seg_pos={0 0}, upload=439093]: Long tail cancellation due to optimal threshold Dec 26 11:46:59.846564475 (500h17m59.71764752s) [noise=true , address=145.79.227.40:4050 , piece_id=SG7WZQGR3BAKWXBJYMTETTPSL3G4CYLR7JGZS6YLWCW7T6SCMKIA, seg_pos={0 0}, upload=439093]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 11:46:59.846583922 (500h17m59.717666968s) [noise=true , address=145.79.227.40:4050 , piece_id=SG7WZQGR3BAKWXBJYMTETTPSL3G4CYLR7JGZS6YLWCW7T6SCMKIA, seg_pos={0 0}, upload=439093]: Long tail cancellation due to optimal threshold Dec 26 11:46:59.846927106 (500h17m59.718010151s) [noise=true , address=45.91.116.4:4940 , piece_id=37LJ66K2WGWCXNXWBJBQOYVUGOKR3XUINJIJWTMJEBCNFBHSFSIA, seg_pos={0 0}, upload=439093]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 11:46:59.846937526 (500h17m59.718020571s) [noise=true , address=45.91.116.4:4940 , piece_id=37LJ66K2WGWCXNXWBJBQOYVUGOKR3XUINJIJWTMJEBCNFBHSFSIA, seg_pos={0 0}, upload=439093]: Long tail cancellation due to optimal threshold Dec 26 11:46:59.846963375 (500h17m59.71804642s) [seg_pos={0 0}, upload=439093]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:46:59.847041654 (500h17m59.718124699s) [upload=439093]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:46:59.847048066 (500h17m59.718131101s) [upload=439093]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:47:00.138103137 (500h18m0.009186182s) [upload=439093]: Done uploading segments... Dec 26 11:47:00.138127123 (500h18m0.009210158s) [upload=439093]: Upload finished. err: <nil> Duration: 6.035671832s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 1.35964111s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 11:47:38 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.359679164s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.X......XX......XXX.XX......X.X..XX.X.X...X....X......X.X.X] duration: 30.00332695s trace: https://jaeger.dev.storj.io/trace/35234c041cadbaf FAIL (2025-12-26 11:47:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 11:46:56.696152839 (500h17m56.567235884s) [upload=439093]: Starting upload Dec 26 11:46:56.696237520 (500h17m56.567320555s) [upload=439093]: Done starting upload Dec 26 11:46:56.696299999 (500h17m56.567383044s) [upload=439093]: Uploading segments... Dec 26 11:46:56.696549154 (500h17m56.567632199s) [upload=439093]: Got next segment. Inline: false Dec 26 11:46:56.696589971 (500h17m56.567673016s) [upload=439093]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:46:56.696596023 (500h17m56.567679068s) [upload=439093]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:46:56.944668216 (500h17m56.815751251s) [seg_pos={0 0}, upload=439093]: Begin upload segment... Dec 26 11:46:56.944719344 (500h17m56.815802379s) [seg_pos={0 0}, upload=439093]: Done begin upload segment. Dec 26 11:46:56.944739571 (500h17m56.815822617s) [noise=true , address=45.91.116.13:4156 , piece_id=QQX3ZH372HRMWSBJUZAFHI2AZOS43RQ7PXHIPWQXHSRYE45US4NA, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.944750413 (500h17m56.815833448s) [noise=true , address=145.79.227.12:4765 , piece_id=3QY2FW65JIAC6GJFJWSQ2MWB6IBPUJXUUVUEJOICFCPG6DZUYMLQ, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.944860813 (500h17m56.815943868s) [noise=true , address=45.91.116.14:4192 , piece_id=4F2T25NTGIJZ7JSXO2T6J5SFQ6FJEH3B4I65QOVFBXE46OHVVA7A, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.945426259 (500h17m56.816509314s) [noise=true , address=145.79.227.45:4231 , piece_id=QY3EL4UBOXHUA3TQZ3NIIWK7MKYHTWJGGE6MHLTLKVX5NIXWU7HQ, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.945675203 (500h17m56.816758268s) [noise=true , address=145.79.227.40:4050 , piece_id=SG7WZQGR3BAKWXBJYMTETTPSL3G4CYLR7JGZS6YLWCW7T6SCMKIA, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.945985935 (500h17m56.81706898s) [noise=true , address=145.79.227.34:4906 , piece_id=PKPLPFOT7IH55544YJFZCJJBQCFB5GXEYLNJB5B43ECU46XPMAUQ, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.946218518 (500h17m56.817301563s) [noise=true , address=145.79.227.17:4971 , piece_id=5JLBY4JSKGBGEUG2PYUDBVX2YL5U6SFXMCCAWYSA2ZGTO2VGFDYQ, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.946344918 (500h17m56.817427953s) [noise=true , address=145.79.227.16:4912 , piece_id=Y7DGGMUOOKBN4KM32QANMLZLDF2ETZ5MYROR5IVJOSAEFUBUAIYA, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.946835313 (500h17m56.817918348s) [noise=true , address=145.79.227.50:4373 , piece_id=7JXLSVS5QLQG6YHTWZ4VRTQ2ATO2JSSNXPT4HIH5MYYJ7PQM6CDA, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.946845582 (500h17m56.817928627s) [noise=true , address=45.91.116.5:4300 , piece_id=TVBREMHSNQHOSXBN3XL6OHRHX3HODOETCWZ2BLZZIFVVSZ4257RQ, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.947250723 (500h17m56.818333768s) [noise=true , address=145.79.227.39:4014 , piece_id=C6H2TJK5HYGJNL3Q7LYZTM2MSKP7FBZJEPGGUOZWXQB42WADBREA, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.947660504 (500h17m56.818743549s) [noise=true , address=145.79.227.20:4439 , piece_id=PRCIGDS7YT7AY6L25LZ4CBQ2PVD4AZMIXMGQ5XXUJZZQ4VTIDP5A, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.948006422 (500h17m56.819089467s) [noise=true , address=45.91.116.4:4940 , piece_id=37LJ66K2WGWCXNXWBJBQOYVUGOKR3XUINJIJWTMJEBCNFBHSFSIA, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.948402587 (500h17m56.819485622s) [noise=true , address=145.79.227.2:4113 , piece_id=TDCOQJ4HLGHZPTAFRN76ZHUS7ZVNBTNO2ZD2CJE6XTJKXZY6FJIQ, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.948793071 (500h17m56.819876106s) [noise=true , address=77.90.31.15:4700 , piece_id=KNTRZ433WA4RGQMPV4TOTXVI4NYEYHHWENJCEZU4ZUU74NFDX3KQ, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:56.949174797 (500h17m56.820257833s) [noise=true , address=145.79.227.51:4411 , piece_id=5H3ZQU5JGUJKUEIPWLEHT7Y2Z6CJBJSPDISJNL77F3ZQVNDOUAUA, seg_pos={0 0}, upload=439093]: Uploading piece... Dec 26 11:46:58.659906146 (500h17m58.530989191s) [upload=439093]: Next returned nil segment Dec 26 11:46:58.659924540 (500h17m58.531007585s) [upload=439093]: Waiting for error group managing segments... Dec 26 11:46:58.943963151 (500h17m58.815046186s) [noise=true , address=145.79.227.17:4971 , piece_id=5JLBY4JSKGBGEUG2PYUDBVX2YL5U6SFXMCCAWYSA2ZGTO2VGFDYQ, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.948494690 (500h17m58.819577725s) [noise=true , address=145.79.227.51:4411 , piece_id=5H3ZQU5JGUJKUEIPWLEHT7Y2Z6CJBJSPDISJNL77F3ZQVNDOUAUA, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.951401525 (500h17m58.82248456s) [noise=true , address=145.79.227.12:4765 , piece_id=3QY2FW65JIAC6GJFJWSQ2MWB6IBPUJXUUVUEJOICFCPG6DZUYMLQ, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.956946003 (500h17m58.828029038s) [noise=true , address=145.79.227.34:4906 , piece_id=PKPLPFOT7IH55544YJFZCJJBQCFB5GXEYLNJB5B43ECU46XPMAUQ, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.957240343 (500h17m58.828323398s) [noise=true , address=145.79.227.20:4439 , piece_id=PRCIGDS7YT7AY6L25LZ4CBQ2PVD4AZMIXMGQ5XXUJZZQ4VTIDP5A, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.957902243 (500h17m58.828985288s) [noise=true , address=145.79.227.2:4113 , piece_id=TDCOQJ4HLGHZPTAFRN76ZHUS7ZVNBTNO2ZD2CJE6XTJKXZY6FJIQ, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.958055986 (500h17m58.829139031s) [noise=true , address=145.79.227.45:4231 , piece_id=QY3EL4UBOXHUA3TQZ3NIIWK7MKYHTWJGGE6MHLTLKVX5NIXWU7HQ, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.959094714 (500h17m58.83017777s) [noise=true , address=145.79.227.39:4014 , piece_id=C6H2TJK5HYGJNL3Q7LYZTM2MSKP7FBZJEPGGUOZWXQB42WADBREA, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.959788295 (500h17m58.83087136s) [noise=true , address=145.79.227.16:4912 , piece_id=Y7DGGMUOOKBN4KM32QANMLZLDF2ETZ5MYROR5IVJOSAEFUBUAIYA, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:58.963958746 (500h17m58.835041791s) [noise=true , address=145.79.227.50:4373 , piece_id=7JXLSVS5QLQG6YHTWZ4VRTQ2ATO2JSSNXPT4HIH5MYYJ7PQM6CDA, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:59.176989905 (500h17m59.04807295s) [noise=true , address=45.91.116.14:4192 , piece_id=4F2T25NTGIJZ7JSXO2T6J5SFQ6FJEH3B4I65QOVFBXE46OHVVA7A, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:59.517256223 (500h17m59.388339268s) [noise=true , address=77.90.31.15:4700 , piece_id=KNTRZ433WA4RGQMPV4TOTXVI4NYEYHHWENJCEZU4ZUU74NFDX3KQ, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:59.846070014 (500h17m59.717153049s) [noise=true , address=45.91.116.13:4156 , piece_id=QQX3ZH372HRMWSBJUZAFHI2AZOS43RQ7PXHIPWQXHSRYE45US4NA, seg_pos={0 0}, upload=439093]: Done uploading piece. err: <nil> Dec 26 11:46:59.846090001 (500h17m59.717173036s) [seg_pos={0 0}, upload=439093]: Segment reached optimal threshold of 13 pieces. Dec 26 11:46:59.846305031 (500h17m59.717388076s) [noise=true , address=45.91.116.5:4300 , piece_id=TVBREMHSNQHOSXBN3XL6OHRHX3HODOETCWZ2BLZZIFVVSZ4257RQ, seg_pos={0 0}, upload=439093]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:55962->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:55962->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55962->45.91.116.5:4300: use of closed network connection Dec 26 11:46:59.846312235 (500h17m59.71739529s) [noise=true , address=45.91.116.5:4300 , piece_id=TVBREMHSNQHOSXBN3XL6OHRHX3HODOETCWZ2BLZZIFVVSZ4257RQ, seg_pos={0 0}, upload=439093]: Long tail cancellation due to optimal threshold Dec 26 11:46:59.846564475 (500h17m59.71764752s) [noise=true , address=145.79.227.40:4050 , piece_id=SG7WZQGR3BAKWXBJYMTETTPSL3G4CYLR7JGZS6YLWCW7T6SCMKIA, seg_pos={0 0}, upload=439093]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 11:46:59.846583922 (500h17m59.717666968s) [noise=true , address=145.79.227.40:4050 , piece_id=SG7WZQGR3BAKWXBJYMTETTPSL3G4CYLR7JGZS6YLWCW7T6SCMKIA, seg_pos={0 0}, upload=439093]: Long tail cancellation due to optimal threshold Dec 26 11:46:59.846927106 (500h17m59.718010151s) [noise=true , address=45.91.116.4:4940 , piece_id=37LJ66K2WGWCXNXWBJBQOYVUGOKR3XUINJIJWTMJEBCNFBHSFSIA, seg_pos={0 0}, upload=439093]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 11:46:59.846937526 (500h17m59.718020571s) [noise=true , address=45.91.116.4:4940 , piece_id=37LJ66K2WGWCXNXWBJBQOYVUGOKR3XUINJIJWTMJEBCNFBHSFSIA, seg_pos={0 0}, upload=439093]: Long tail cancellation due to optimal threshold Dec 26 11:46:59.846963375 (500h17m59.71804642s) [seg_pos={0 0}, upload=439093]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:46:59.847041654 (500h17m59.718124699s) [upload=439093]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:46:59.847048066 (500h17m59.718131101s) [upload=439093]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:47:00.138103137 (500h18m0.009186182s) [upload=439093]: Done uploading segments... Dec 26 11:47:00.138127123 (500h18m0.009210158s) [upload=439093]: Upload finished. err: <nil> Duration: 30.003357156s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................................X....X....] duration: 3.372148257s trace: https://jaeger.dev.storj.io/trace/10a24935a8f16015 FAIL (2025-12-26 11:46:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.372174526s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.363294967s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 11:46:43 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.363322731s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 1.39665404s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 11:46:03 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.39668574s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X.X......XX......XXX.XX......X.X..XX.X.X...X....X......X.X.] duration: 5.768531798s trace: https://jaeger.dev.storj.io/trace/47c650fb77053183 FAIL (2025-12-26 11:45:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 11:44:48.595233053 (500h15m48.466316088s) [upload=439066]: Starting upload Dec 26 11:44:48.595313035 (500h15m48.46639607s) [upload=439066]: Done starting upload Dec 26 11:44:48.595394421 (500h15m48.466477456s) [upload=439066]: Uploading segments... Dec 26 11:44:48.595695804 (500h15m48.466778839s) [upload=439066]: Got next segment. Inline: false Dec 26 11:44:48.595737183 (500h15m48.466820218s) [upload=439066]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:44:48.595741913 (500h15m48.466824948s) [upload=439066]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:44:48.845422359 (500h15m48.716505394s) [seg_pos={0 0}, upload=439066]: Begin upload segment... Dec 26 11:44:48.845481963 (500h15m48.716564998s) [seg_pos={0 0}, upload=439066]: Done begin upload segment. Dec 26 11:44:48.845626207 (500h15m48.716709242s) [noise=true , address=77.90.31.18:4805 , piece_id=5EX5FXYAFIADXIC5FJVTUP2OFXFIFBB3Q4WLEVNKBXEEIKTUDPYQ, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.845646295 (500h15m48.71672933s) [noise=true , address=145.79.227.39:4015 , piece_id=NBYTLTFLJYP7U4DZAHH2EWN5C26RUYDOANOG7O645VGJ6PS67Y3Q, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.846793439 (500h15m48.717876474s) [noise=true , address=45.91.116.11:4119 , piece_id=4YSDKMQQGEEIVAO4S6JF4QTXWXNYAA5AZ5LA7P5IZDTJGKWZCVMA, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.846944227 (500h15m48.718027262s) [noise=true , address=145.79.227.51:4409 , piece_id=L65G3325NQ24FH7DHVWUJA5A73LC4EFFBSNOFZHPKPBQJSKCPVUA, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.847657755 (500h15m48.71874079s) [noise=true , address=145.79.227.45:4231 , piece_id=JDVJUIRS7XW7L2454N64FYIKMHHAD6KPQ7AOL3P4PMO5G357N2MA, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.847683134 (500h15m48.718766169s) [noise=true , address=145.79.227.17:4970 , piece_id=5GSE5TG2D2G7EP4DLVW2BWM3EZAKFSF4HXMEBTMGJ37NLXH7ADYQ, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.848164641 (500h15m48.719247676s) [noise=true , address=145.79.227.31:4799 , piece_id=MVHPMMNB5WM3FTTKTIK52IBREW6WAHI6OTEXQLWWDQQIS35W4COQ, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.848223423 (500h15m48.719306468s) [noise=true , address=145.79.227.26:4654 , piece_id=WBBTZYQJPHJW57DCDV4XXSBVWT6LAU4RWVFAJZYWYN6USNN2GRAQ, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.848661266 (500h15m48.719744311s) [noise=true , address=145.79.227.38:4979 , piece_id=RKE6MDC354R3IH4O4W3HIQJ7OT4H3DEP7TYBIL2JQO7AMJPTYTSA, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.848930459 (500h15m48.720013494s) [noise=true , address=45.91.116.7:4012 , piece_id=B4HT6GVER7HHWHJR7DL6VWOSPZ7K35H27NWV6HUZVJVWR7X66AAQ, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.849086125 (500h15m48.72016917s) [noise=true , address=77.90.31.10:4517 , piece_id=XUGWAQ3JOIHOKYGNV3PD72RGJ6VIAZCEOEKYFKOHEVQKJXQVDJMA, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.849449938 (500h15m48.720532983s) [noise=true , address=77.90.31.14:4663 , piece_id=RZATZWHEBHI6FADSEPJJ4LZTXZZ7CKNZLPVBS45DASNX7QJCXRGQ, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.849588282 (500h15m48.720671327s) [noise=true , address=145.79.227.19:4077 , piece_id=Z3LHQHCNIS4WKLQKUOPP45BDZPG65CVBCU6QKQNGGPU73VLKRRYA, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.850151083 (500h15m48.721234128s) [noise=true , address=145.79.227.3:4151 , piece_id=VEML2GR2HWPZSWB7KM2CZXADQIWCUUROUWYCJW2JVHFFQAUBQXYQ, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.850816320 (500h15m48.721899365s) [noise=true , address=145.79.227.20:4439 , piece_id=P2L3RJWE44LROEQHMCSXUZO6LGNVTPXCUI4S3OJVREYAL6LOFN7Q, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.852576282 (500h15m48.723659317s) [noise=true , address=145.79.227.43:4157 , piece_id=XRLWV6QJQCDKRNGH4HP2RT5XPWFZDNCRODPZU5OFJNCLSVG6ULAA, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:49.994199206 (500h15m49.865282251s) [noise=true , address=77.90.31.18:4805 , piece_id=5EX5FXYAFIADXIC5FJVTUP2OFXFIFBB3Q4WLEVNKBXEEIKTUDPYQ, seg_pos={0 0}, upload=439066]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 11:44:50.563878547 (500h15m50.434961593s) [upload=439066]: Next returned nil segment Dec 26 11:44:50.563933862 (500h15m50.435016967s) [upload=439066]: Waiting for error group managing segments... Dec 26 11:44:50.851046867 (500h15m50.722129902s) [noise=true , address=145.79.227.31:4799 , piece_id=MVHPMMNB5WM3FTTKTIK52IBREW6WAHI6OTEXQLWWDQQIS35W4COQ, seg_pos={0 0}, upload=439066]: Done uploading piece. err: <nil> Dec 26 11:44:50.853468429 (500h15m50.724551464s) [noise=true , address=145.79.227.19:4077 , piece_id=Z3LHQHCNIS4WKLQKUOPP45BDZPG65CVBCU6QKQNGGPU73VLKRRYA, seg_pos={0 0}, upload=439066]: Done uploading piece. err: <nil> Dec 26 11:44:50.857085666 (500h15m50.728168721s) [noise=true , address=145.79.227.39:4015 , piece_id=NBYTLTFLJYP7U4DZAHH2EWN5C26RUYDOANOG7O645VGJ6PS67Y3Q, seg_pos={0 0}, upload=439066]: Done uploading piece. err: <nil> Dec 26 11:44:50.857919914 (500h15m50.729002949s) [noise=true , address=145.79.227.51:4409 , piece_id=L65G3325NQ24FH7DHVWUJA5A73LC4EFFBSNOFZHPKPBQJSKCPVUA, seg_pos={0 0}, upload=439066]: Done uploading piece. err: <nil> Dec 26 11:44:50.871878283 (500h15m50.742961328s) [noise=true , address=145.79.227.45:4231 , piece_id=JDVJUIRS7XW7L2454N64FYIKMHHAD6KPQ7AOL3P4PMO5G357N2MA, seg_pos={0 0}, upload=439066]: Done uploading piece. err: <nil> Dec 26 11:44:50.879023089 (500h15m50.750106134s) [noise=true , address=145.79.227.38:4979 , piece_id=RKE6MDC354R3IH4O4W3HIQJ7OT4H3DEP7TYBIL2JQO7AMJPTYTSA, seg_pos={0 0}, upload=439066]: Done uploading piece. err: <nil> Dec 26 11:44:50.883475567 (500h15m50.754558602s) [noise=true , address=145.79.227.17:4970 , piece_id=5GSE5TG2D2G7EP4DLVW2BWM3EZAKFSF4HXMEBTMGJ37NLXH7ADYQ, seg_pos={0 0}, upload=439066]: Done uploading piece. err: <nil> Dec 26 11:44:50.886858860 (500h15m50.757941905s) [noise=true , address=145.79.227.20:4439 , piece_id=P2L3RJWE44LROEQHMCSXUZO6LGNVTPXCUI4S3OJVREYAL6LOFN7Q, seg_pos={0 0}, upload=439066]: Done uploading piece. err: <nil> Dec 26 11:44:50.901894560 (500h15m50.772977595s) [noise=true , address=145.79.227.26:4654 , piece_id=WBBTZYQJPHJW57DCDV4XXSBVWT6LAU4RWVFAJZYWYN6USNN2GRAQ, seg_pos={0 0}, upload=439066]: Done uploading piece. err: <nil> Dec 26 11:44:50.950802780 (500h15m50.821885815s) [noise=true , address=145.79.227.43:4157 , piece_id=XRLWV6QJQCDKRNGH4HP2RT5XPWFZDNCRODPZU5OFJNCLSVG6ULAA, seg_pos={0 0}, upload=439066]: Done uploading piece. err: <nil> Dec 26 11:44:51.052950006 (500h15m50.924033072s) [noise=true , address=145.79.227.3:4151 , piece_id=VEML2GR2HWPZSWB7KM2CZXADQIWCUUROUWYCJW2JVHFFQAUBQXYQ, seg_pos={0 0}, upload=439066]: Done uploading piece. err: <nil> Dec 26 11:44:51.143349634 (500h15m51.014432669s) [noise=true , address=45.91.116.7:4012 , piece_id=B4HT6GVER7HHWHJR7DL6VWOSPZ7K35H27NWV6HUZVJVWR7X66AAQ, seg_pos={0 0}, upload=439066]: Done uploading piece. err: <nil> Dec 26 11:44:51.598399120 (500h15m51.469482155s) [noise=true , address=45.91.116.11:4119 , piece_id=4YSDKMQQGEEIVAO4S6JF4QTXWXNYAA5AZ5LA7P5IZDTJGKWZCVMA, seg_pos={0 0}, upload=439066]: Done uploading piece. err: <nil> Dec 26 11:44:51.598428005 (500h15m51.46951104s) [seg_pos={0 0}, upload=439066]: Segment reached optimal threshold of 13 pieces. Dec 26 11:44:51.598827727 (500h15m51.469910792s) [noise=true , address=77.90.31.10:4517 , piece_id=XUGWAQ3JOIHOKYGNV3PD72RGJ6VIAZCEOEKYFKOHEVQKJXQVDJMA, seg_pos={0 0}, upload=439066]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:37984->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:37984->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37984->77.90.31.10:4517: use of closed network connection Dec 26 11:44:51.598837876 (500h15m51.469920921s) [noise=true , address=77.90.31.10:4517 , piece_id=XUGWAQ3JOIHOKYGNV3PD72RGJ6VIAZCEOEKYFKOHEVQKJXQVDJMA, seg_pos={0 0}, upload=439066]: Long tail cancellation due to optimal threshold Dec 26 11:44:51.598901607 (500h15m51.469984642s) [noise=true , address=77.90.31.14:4663 , piece_id=RZATZWHEBHI6FADSEPJJ4LZTXZZ7CKNZLPVBS45DASNX7QJCXRGQ, seg_pos={0 0}, upload=439066]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:56492->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:56492->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56492->77.90.31.14:4663: use of closed network connection Dec 26 11:44:51.598917146 (500h15m51.470000181s) [noise=true , address=77.90.31.14:4663 , piece_id=RZATZWHEBHI6FADSEPJJ4LZTXZZ7CKNZLPVBS45DASNX7QJCXRGQ, seg_pos={0 0}, upload=439066]: Long tail cancellation due to optimal threshold Dec 26 11:44:51.598940852 (500h15m51.470023887s) [seg_pos={0 0}, upload=439066]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:44:51.599027677 (500h15m51.470110712s) [upload=439066]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:44:51.599032586 (500h15m51.470115621s) [upload=439066]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:44:51.887295651 (500h15m51.758378696s) [upload=439066]: Done uploading segments... Dec 26 11:44:51.887310669 (500h15m51.758393714s) [upload=439066]: Upload finished. err: <nil> Duration: 5.768561756s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................................................X....X...] duration: 4.67069589s trace: https://jaeger.dev.storj.io/trace/10a24935a8f16015 FAIL (2025-12-26 11:45:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.670723282s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X.X......XX......XXX.XX......X.X..XX.X.X...X....X......X.X] duration: 30.003285464s trace: https://jaeger.dev.storj.io/trace/47c650fb77053183 FAIL (2025-12-26 11:45:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 11:44:48.595233053 (500h15m48.466316088s) [upload=439066]: Starting upload Dec 26 11:44:48.595313035 (500h15m48.46639607s) [upload=439066]: Done starting upload Dec 26 11:44:48.595394421 (500h15m48.466477456s) [upload=439066]: Uploading segments... Dec 26 11:44:48.595695804 (500h15m48.466778839s) [upload=439066]: Got next segment. Inline: false Dec 26 11:44:48.595737183 (500h15m48.466820218s) [upload=439066]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:44:48.595741913 (500h15m48.466824948s) [upload=439066]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:44:48.845422359 (500h15m48.716505394s) [seg_pos={0 0}, upload=439066]: Begin upload segment... Dec 26 11:44:48.845481963 (500h15m48.716564998s) [seg_pos={0 0}, upload=439066]: Done begin upload segment. Dec 26 11:44:48.845626207 (500h15m48.716709242s) [noise=true , address=77.90.31.18:4805 , piece_id=5EX5FXYAFIADXIC5FJVTUP2OFXFIFBB3Q4WLEVNKBXEEIKTUDPYQ, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.845646295 (500h15m48.71672933s) [noise=true , address=145.79.227.39:4015 , piece_id=NBYTLTFLJYP7U4DZAHH2EWN5C26RUYDOANOG7O645VGJ6PS67Y3Q, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.846793439 (500h15m48.717876474s) [noise=true , address=45.91.116.11:4119 , piece_id=4YSDKMQQGEEIVAO4S6JF4QTXWXNYAA5AZ5LA7P5IZDTJGKWZCVMA, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.846944227 (500h15m48.718027262s) [noise=true , address=145.79.227.51:4409 , piece_id=L65G3325NQ24FH7DHVWUJA5A73LC4EFFBSNOFZHPKPBQJSKCPVUA, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.847657755 (500h15m48.71874079s) [noise=true , address=145.79.227.45:4231 , piece_id=JDVJUIRS7XW7L2454N64FYIKMHHAD6KPQ7AOL3P4PMO5G357N2MA, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.847683134 (500h15m48.718766169s) [noise=true , address=145.79.227.17:4970 , piece_id=5GSE5TG2D2G7EP4DLVW2BWM3EZAKFSF4HXMEBTMGJ37NLXH7ADYQ, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.848164641 (500h15m48.719247676s) [noise=true , address=145.79.227.31:4799 , piece_id=MVHPMMNB5WM3FTTKTIK52IBREW6WAHI6OTEXQLWWDQQIS35W4COQ, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.848223423 (500h15m48.719306468s) [noise=true , address=145.79.227.26:4654 , piece_id=WBBTZYQJPHJW57DCDV4XXSBVWT6LAU4RWVFAJZYWYN6USNN2GRAQ, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.848661266 (500h15m48.719744311s) [noise=true , address=145.79.227.38:4979 , piece_id=RKE6MDC354R3IH4O4W3HIQJ7OT4H3DEP7TYBIL2JQO7AMJPTYTSA, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.848930459 (500h15m48.720013494s) [noise=true , address=45.91.116.7:4012 , piece_id=B4HT6GVER7HHWHJR7DL6VWOSPZ7K35H27NWV6HUZVJVWR7X66AAQ, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.849086125 (500h15m48.72016917s) [noise=true , address=77.90.31.10:4517 , piece_id=XUGWAQ3JOIHOKYGNV3PD72RGJ6VIAZCEOEKYFKOHEVQKJXQVDJMA, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.849449938 (500h15m48.720532983s) [noise=true , address=77.90.31.14:4663 , piece_id=RZATZWHEBHI6FADSEPJJ4LZTXZZ7CKNZLPVBS45DASNX7QJCXRGQ, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.849588282 (500h15m48.720671327s) [noise=true , address=145.79.227.19:4077 , piece_id=Z3LHQHCNIS4WKLQKUOPP45BDZPG65CVBCU6QKQNGGPU73VLKRRYA, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.850151083 (500h15m48.721234128s) [noise=true , address=145.79.227.3:4151 , piece_id=VEML2GR2HWPZSWB7KM2CZXADQIWCUUROUWYCJW2JVHFFQAUBQXYQ, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.850816320 (500h15m48.721899365s) [noise=true , address=145.79.227.20:4439 , piece_id=P2L3RJWE44LROEQHMCSXUZO6LGNVTPXCUI4S3OJVREYAL6LOFN7Q, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:48.852576282 (500h15m48.723659317s) [noise=true , address=145.79.227.43:4157 , piece_id=XRLWV6QJQCDKRNGH4HP2RT5XPWFZDNCRODPZU5OFJNCLSVG6ULAA, seg_pos={0 0}, upload=439066]: Uploading piece... Dec 26 11:44:49.994199206 (500h15m49.865282251s) [noise=true , address=77.90.31.18:4805 , piece_id=5EX5FXYAFIADXIC5FJVTUP2OFXFIFBB3Q4WLEVNKBXEEIKTUDPYQ, seg_pos={0 0}, upload=439066]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 11:44:50.563878547 (500h15m50.434961593s) [upload=439066]: Next returned nil segment Dec 26 11:44:50.563933862 (500h15m50.435016967s) [upload=439066]: Waiting for error group managing segments... Dec 26 11:44:50.851046867 (500h15m50.722129902s) [noise=true , address=145.79.227.31:4799 , piece_id=MVHPMMNB5WM3FTTKTIK52IBREW6WAHI6OTEXQLWWDQQIS35W4COQ, seg_pos={0 0}, upload=439066]: Done uploading piece. err: <nil> Dec 26 11:44:50.853468429 (500h15m50.724551464s) [noise=true , address=145.79.227.19:4077 , piece_id=Z3LHQHCNIS4WKLQKUOPP45BDZPG65CVBCU6QKQNGGPU73VLKRRYA, seg_pos={0 0}, upload=439066]: Done uploading piece. err: <nil> Dec 26 11:44:50.857085666 (500h15m50.728168721s) [noise=true , address=145.79.227.39:4015 , piece_id=NBYTLTFLJYP7U4DZAHH2EWN5C26RUYDOANOG7O645VGJ6PS67Y3Q, seg_pos={0 0}, upload=439066]: Done uploading piece. err: <nil> Dec 26 11:44:50.857919914 (500h15m50.729002949s) [noise=true , address=145.79.227.51:4409 , piece_id=L65G3325NQ24FH7DHVWUJA5A73LC4EFFBSNOFZHPKPBQJSKCPVUA, seg_pos={0 0}, upload=439066]: Done uploading piece. err: <nil> Dec 26 11:44:50.871878283 (500h15m50.742961328s) [noise=true , address=145.79.227.45:4231 , piece_id=JDVJUIRS7XW7L2454N64FYIKMHHAD6KPQ7AOL3P4PMO5G357N2MA, seg_pos={0 0}, upload=439066]: Done uploading piece. err: <nil> Dec 26 11:44:50.879023089 (500h15m50.750106134s) [noise=true , address=145.79.227.38:4979 , piece_id=RKE6MDC354R3IH4O4W3HIQJ7OT4H3DEP7TYBIL2JQO7AMJPTYTSA, seg_pos={0 0}, upload=439066]: Done uploading piece. err: <nil> Dec 26 11:44:50.883475567 (500h15m50.754558602s) [noise=true , address=145.79.227.17:4970 , piece_id=5GSE5TG2D2G7EP4DLVW2BWM3EZAKFSF4HXMEBTMGJ37NLXH7ADYQ, seg_pos={0 0}, upload=439066]: Done uploading piece. err: <nil> Dec 26 11:44:50.886858860 (500h15m50.757941905s) [noise=true , address=145.79.227.20:4439 , piece_id=P2L3RJWE44LROEQHMCSXUZO6LGNVTPXCUI4S3OJVREYAL6LOFN7Q, seg_pos={0 0}, upload=439066]: Done uploading piece. err: <nil> Dec 26 11:44:50.901894560 (500h15m50.772977595s) [noise=true , address=145.79.227.26:4654 , piece_id=WBBTZYQJPHJW57DCDV4XXSBVWT6LAU4RWVFAJZYWYN6USNN2GRAQ, seg_pos={0 0}, upload=439066]: Done uploading piece. err: <nil> Dec 26 11:44:50.950802780 (500h15m50.821885815s) [noise=true , address=145.79.227.43:4157 , piece_id=XRLWV6QJQCDKRNGH4HP2RT5XPWFZDNCRODPZU5OFJNCLSVG6ULAA, seg_pos={0 0}, upload=439066]: Done uploading piece. err: <nil> Dec 26 11:44:51.052950006 (500h15m50.924033072s) [noise=true , address=145.79.227.3:4151 , piece_id=VEML2GR2HWPZSWB7KM2CZXADQIWCUUROUWYCJW2JVHFFQAUBQXYQ, seg_pos={0 0}, upload=439066]: Done uploading piece. err: <nil> Dec 26 11:44:51.143349634 (500h15m51.014432669s) [noise=true , address=45.91.116.7:4012 , piece_id=B4HT6GVER7HHWHJR7DL6VWOSPZ7K35H27NWV6HUZVJVWR7X66AAQ, seg_pos={0 0}, upload=439066]: Done uploading piece. err: <nil> Dec 26 11:44:51.598399120 (500h15m51.469482155s) [noise=true , address=45.91.116.11:4119 , piece_id=4YSDKMQQGEEIVAO4S6JF4QTXWXNYAA5AZ5LA7P5IZDTJGKWZCVMA, seg_pos={0 0}, upload=439066]: Done uploading piece. err: <nil> Dec 26 11:44:51.598428005 (500h15m51.46951104s) [seg_pos={0 0}, upload=439066]: Segment reached optimal threshold of 13 pieces. Dec 26 11:44:51.598827727 (500h15m51.469910792s) [noise=true , address=77.90.31.10:4517 , piece_id=XUGWAQ3JOIHOKYGNV3PD72RGJ6VIAZCEOEKYFKOHEVQKJXQVDJMA, seg_pos={0 0}, upload=439066]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:37984->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:37984->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37984->77.90.31.10:4517: use of closed network connection Dec 26 11:44:51.598837876 (500h15m51.469920921s) [noise=true , address=77.90.31.10:4517 , piece_id=XUGWAQ3JOIHOKYGNV3PD72RGJ6VIAZCEOEKYFKOHEVQKJXQVDJMA, seg_pos={0 0}, upload=439066]: Long tail cancellation due to optimal threshold Dec 26 11:44:51.598901607 (500h15m51.469984642s) [noise=true , address=77.90.31.14:4663 , piece_id=RZATZWHEBHI6FADSEPJJ4LZTXZZ7CKNZLPVBS45DASNX7QJCXRGQ, seg_pos={0 0}, upload=439066]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:56492->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:56492->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56492->77.90.31.14:4663: use of closed network connection Dec 26 11:44:51.598917146 (500h15m51.470000181s) [noise=true , address=77.90.31.14:4663 , piece_id=RZATZWHEBHI6FADSEPJJ4LZTXZZ7CKNZLPVBS45DASNX7QJCXRGQ, seg_pos={0 0}, upload=439066]: Long tail cancellation due to optimal threshold Dec 26 11:44:51.598940852 (500h15m51.470023887s) [seg_pos={0 0}, upload=439066]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:44:51.599027677 (500h15m51.470110712s) [upload=439066]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:44:51.599032586 (500h15m51.470115621s) [upload=439066]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:44:51.887295651 (500h15m51.758378696s) [upload=439066]: Done uploading segments... Dec 26 11:44:51.887310669 (500h15m51.758393714s) [upload=439066]: Upload finished. err: <nil> Duration: 30.003307414s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 1.400360898s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 11:45:03 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.400409911s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................................................X....X..] duration: 3.514789859s trace: https://jaeger.dev.storj.io/trace/10a24935a8f16015 FAIL (2025-12-26 11:44:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.514812932s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 1.440102134s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 11:44:47 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.440127902s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................................................X....X.] duration: 3.732350523s trace: https://jaeger.dev.storj.io/trace/10a24935a8f16015 FAIL (2025-12-26 11:43:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.732375751s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 1.427080549s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 11:43:51 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.427111387s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X..X.X......XX......XXX.XX......X.X..XX.X.X...X....X......X.] duration: 9.720112985s trace: https://jaeger.dev.storj.io/trace/3fde6f1f29d4ccdf FAIL (2025-12-26 11:43:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 11:43:02.694115477 (500h14m2.565198522s) [upload=439041]: Starting upload Dec 26 11:43:02.694212142 (500h14m2.565295197s) [upload=439041]: Done starting upload Dec 26 11:43:02.694266155 (500h14m2.56534921s) [upload=439041]: Uploading segments... Dec 26 11:43:02.694616532 (500h14m2.565699577s) [upload=439041]: Got next segment. Inline: false Dec 26 11:43:02.694752922 (500h14m2.565835967s) [upload=439041]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:43:02.694761097 (500h14m2.565844162s) [upload=439041]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:43:02.943375264 (500h14m2.814458309s) [seg_pos={0 0}, upload=439041]: Begin upload segment... Dec 26 11:43:02.943442140 (500h14m2.814525185s) [seg_pos={0 0}, upload=439041]: Done begin upload segment. Dec 26 11:43:02.943481626 (500h14m2.814564671s) [noise=true , address=145.79.227.46:4603 , piece_id=QGECIFBCGTVK4BFAADEPB7AN4CEEV3YQEQAG7XVLX6PTUA6ZCYHA, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.943586476 (500h14m2.814669521s) [noise=true , address=45.91.116.12:4226 , piece_id=IJVZGCOZCQNJ7XPZ2X5ZAOBJ44UXAPC4EP5DBPILBWBTOGSQE2CA, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.944283653 (500h14m2.815366698s) [noise=true , address=77.90.31.19:4844 , piece_id=47PJR5G7ZUXBKUDSAYV2IEP44ILGVE4TTOPERLNOKI6II56SC3WQ, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.944746334 (500h14m2.815829379s) [noise=true , address=45.91.116.10:4084 , piece_id=A3F7N5VC7PKUHOPCMFJDLXRC66JWAJUSBVPHCKWXUD3DN3QS4U2A, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.945155062 (500h14m2.816238108s) [noise=true , address=77.90.31.14:4664 , piece_id=UV5FJNAFSA24B3UV7DD2FOIZBN6Q6SGJVUFOW6JL42JTLYCQ6QPA, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.945577507 (500h14m2.816660552s) [noise=true , address=145.79.227.43:4157 , piece_id=3AGZ7W3LG5BHYCCYYBKABOIHDIX62JKEOW622MNPR2X522GMQMTA, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.946013176 (500h14m2.817096221s) [noise=true , address=145.79.227.17:4971 , piece_id=64K7OBJEFCUXFQYHMFHAWQYYR36O74AQYSU7EFR463A6ZBBGYYDQ, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.946353334 (500h14m2.817436379s) [noise=true , address=45.91.116.8:4046 , piece_id=XZSJRP3GUVBX6TQA32S5VCF6ZZAY2SPCVOKFJS7GQWFCJMOWEV3A, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.946403359 (500h14m2.817486404s) [noise=true , address=45.91.116.5:4298 , piece_id=YQLNYJG6A5VRGEIKEVSUSWJ2SK5DDN53XOLHDPSC6IPVDIFMPUNA, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.946819962 (500h14m2.817903007s) [noise=true , address=145.79.227.52:4447 , piece_id=HRYRMA3VE23CSJ3FRDOBUYRVDA4NKP62GL5D3JK3463PUZF6LD3Q, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.947171472 (500h14m2.818254527s) [noise=true , address=145.79.227.34:4907 , piece_id=CMYXL6HPRP36RBKLYVLSDKRRDEBPIVXEDY4ME3PL5X2RS7ZCZM2Q, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.947194876 (500h14m2.818277921s) [noise=true , address=145.79.227.29:4943 , piece_id=PZCHZEHOE2WIWLMCXGLOUZAKIL77EREQBEAIMXTEFLY5Y3H74VYA, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.947677044 (500h14m2.818760089s) [noise=true , address=45.91.116.9:4264 , piece_id=TCT6LOQMSVHCGO66L4XHM6SHICJMUCUARVWJX2TODFFJMB2WXNGA, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.948225429 (500h14m2.819308544s) [noise=true , address=145.79.227.51:4409 , piece_id=H62IFOSSQTEBNTFODIP3DSSRYQXETW57WN7SK642L5NEH6BVX2VQ, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.948790925 (500h14m2.81987397s) [noise=true , address=45.91.116.6:4974 , piece_id=RLXG7YTNDUIKLKRZZULTEJR66VORCITLDTU4PNHVMEHGDWSRCXTA, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.949259729 (500h14m2.820342774s) [noise=true , address=45.91.116.13:4154 , piece_id=5O4LSWDMZIWHC7Q6SBIS76SYGKQL6WMOMZETCKANUT2EPBBGIADQ, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:04.651618675 (500h14m4.52270172s) [upload=439041]: Next returned nil segment Dec 26 11:43:04.651644063 (500h14m4.522727118s) [upload=439041]: Waiting for error group managing segments... Dec 26 11:43:04.939927075 (500h14m4.81101011s) [noise=true , address=145.79.227.46:4603 , piece_id=QGECIFBCGTVK4BFAADEPB7AN4CEEV3YQEQAG7XVLX6PTUA6ZCYHA, seg_pos={0 0}, upload=439041]: Done uploading piece. err: <nil> Dec 26 11:43:04.958907327 (500h14m4.829990372s) [noise=true , address=145.79.227.52:4447 , piece_id=HRYRMA3VE23CSJ3FRDOBUYRVDA4NKP62GL5D3JK3463PUZF6LD3Q, seg_pos={0 0}, upload=439041]: Done uploading piece. err: <nil> Dec 26 11:43:04.959497801 (500h14m4.830580846s) [noise=true , address=145.79.227.29:4943 , piece_id=PZCHZEHOE2WIWLMCXGLOUZAKIL77EREQBEAIMXTEFLY5Y3H74VYA, seg_pos={0 0}, upload=439041]: Done uploading piece. err: <nil> Dec 26 11:43:04.970918169 (500h14m4.842001214s) [noise=true , address=145.79.227.17:4971 , piece_id=64K7OBJEFCUXFQYHMFHAWQYYR36O74AQYSU7EFR463A6ZBBGYYDQ, seg_pos={0 0}, upload=439041]: Done uploading piece. err: <nil> Dec 26 11:43:04.976226957 (500h14m4.847309992s) [noise=true , address=145.79.227.43:4157 , piece_id=3AGZ7W3LG5BHYCCYYBKABOIHDIX62JKEOW622MNPR2X522GMQMTA, seg_pos={0 0}, upload=439041]: Done uploading piece. err: <nil> Dec 26 11:43:04.979618925 (500h14m4.85070197s) [noise=true , address=145.79.227.51:4409 , piece_id=H62IFOSSQTEBNTFODIP3DSSRYQXETW57WN7SK642L5NEH6BVX2VQ, seg_pos={0 0}, upload=439041]: Done uploading piece. err: <nil> Dec 26 11:43:05.186340601 (500h14m5.057423646s) [noise=true , address=45.91.116.6:4974 , piece_id=RLXG7YTNDUIKLKRZZULTEJR66VORCITLDTU4PNHVMEHGDWSRCXTA, seg_pos={0 0}, upload=439041]: Done uploading piece. err: <nil> Dec 26 11:43:05.195561289 (500h14m5.066644344s) [noise=true , address=45.91.116.8:4046 , piece_id=XZSJRP3GUVBX6TQA32S5VCF6ZZAY2SPCVOKFJS7GQWFCJMOWEV3A, seg_pos={0 0}, upload=439041]: Done uploading piece. err: <nil> Dec 26 11:43:05.233599741 (500h14m5.104682786s) [noise=true , address=45.91.116.12:4226 , piece_id=IJVZGCOZCQNJ7XPZ2X5ZAOBJ44UXAPC4EP5DBPILBWBTOGSQE2CA, seg_pos={0 0}, upload=439041]: Done uploading piece. err: <nil> Dec 26 11:43:05.249595820 (500h14m5.120678865s) [noise=true , address=77.90.31.14:4664 , piece_id=UV5FJNAFSA24B3UV7DD2FOIZBN6Q6SGJVUFOW6JL42JTLYCQ6QPA, seg_pos={0 0}, upload=439041]: Done uploading piece. err: <nil> Dec 26 11:43:05.280396631 (500h14m5.151479676s) [noise=true , address=45.91.116.13:4154 , piece_id=5O4LSWDMZIWHC7Q6SBIS76SYGKQL6WMOMZETCKANUT2EPBBGIADQ, seg_pos={0 0}, upload=439041]: Done uploading piece. err: <nil> Dec 26 11:43:05.376228501 (500h14m5.247311536s) [noise=true , address=45.91.116.5:4298 , piece_id=YQLNYJG6A5VRGEIKEVSUSWJ2SK5DDN53XOLHDPSC6IPVDIFMPUNA, seg_pos={0 0}, upload=439041]: Done uploading piece. err: <nil> Dec 26 11:43:05.377105712 (500h14m5.248188747s) [noise=true , address=45.91.116.10:4084 , piece_id=A3F7N5VC7PKUHOPCMFJDLXRC66JWAJUSBVPHCKWXUD3DN3QS4U2A, seg_pos={0 0}, upload=439041]: Done uploading piece. err: <nil> Dec 26 11:43:05.377116212 (500h14m5.248199247s) [seg_pos={0 0}, upload=439041]: Segment reached optimal threshold of 13 pieces. Dec 26 11:43:05.377299871 (500h14m5.248382906s) [noise=true , address=145.79.227.34:4907 , piece_id=CMYXL6HPRP36RBKLYVLSDKRRDEBPIVXEDY4ME3PL5X2RS7ZCZM2Q, seg_pos={0 0}, upload=439041]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 11:43:05.377306514 (500h14m5.248389549s) [noise=true , address=145.79.227.34:4907 , piece_id=CMYXL6HPRP36RBKLYVLSDKRRDEBPIVXEDY4ME3PL5X2RS7ZCZM2Q, seg_pos={0 0}, upload=439041]: Long tail cancellation due to optimal threshold Dec 26 11:43:05.377570898 (500h14m5.248653933s) [noise=true , address=77.90.31.19:4844 , piece_id=47PJR5G7ZUXBKUDSAYV2IEP44ILGVE4TTOPERLNOKI6II56SC3WQ, seg_pos={0 0}, upload=439041]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: rpc: dial tcp 77.90.31.19:4844: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 11:43:05.377577220 (500h14m5.248660255s) [noise=true , address=77.90.31.19:4844 , piece_id=47PJR5G7ZUXBKUDSAYV2IEP44ILGVE4TTOPERLNOKI6II56SC3WQ, seg_pos={0 0}, upload=439041]: Long tail cancellation due to optimal threshold Dec 26 11:43:05.378051663 (500h14m5.249134708s) [noise=true , address=45.91.116.9:4264 , piece_id=TCT6LOQMSVHCGO66L4XHM6SHICJMUCUARVWJX2TODFFJMB2WXNGA, seg_pos={0 0}, upload=439041]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: read tcp 188.166.225.170:49576->45.91.116.9:4264: use of closed network connection; context canceled Dec 26 11:43:05.378072653 (500h14m5.249155698s) [noise=true , address=45.91.116.9:4264 , piece_id=TCT6LOQMSVHCGO66L4XHM6SHICJMUCUARVWJX2TODFFJMB2WXNGA, seg_pos={0 0}, upload=439041]: Long tail cancellation due to optimal threshold Dec 26 11:43:05.378109363 (500h14m5.249192408s) [seg_pos={0 0}, upload=439041]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:43:05.378219383 (500h14m5.249302418s) [upload=439041]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:43:05.378226055 (500h14m5.2493091s) [upload=439041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:43:05.673634416 (500h14m5.544717461s) [upload=439041]: Done uploading segments... Dec 26 11:43:05.673654775 (500h14m5.54473781s) [upload=439041]: Upload finished. err: <nil> Duration: 9.720142803s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X..X.X......XX......XXX.XX......X.X..XX.X.X...X....X......X] duration: 30.003750346s trace: https://jaeger.dev.storj.io/trace/3fde6f1f29d4ccdf FAIL (2025-12-26 11:43:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 11:43:02.694115477 (500h14m2.565198522s) [upload=439041]: Starting upload Dec 26 11:43:02.694212142 (500h14m2.565295197s) [upload=439041]: Done starting upload Dec 26 11:43:02.694266155 (500h14m2.56534921s) [upload=439041]: Uploading segments... Dec 26 11:43:02.694616532 (500h14m2.565699577s) [upload=439041]: Got next segment. Inline: false Dec 26 11:43:02.694752922 (500h14m2.565835967s) [upload=439041]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:43:02.694761097 (500h14m2.565844162s) [upload=439041]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:43:02.943375264 (500h14m2.814458309s) [seg_pos={0 0}, upload=439041]: Begin upload segment... Dec 26 11:43:02.943442140 (500h14m2.814525185s) [seg_pos={0 0}, upload=439041]: Done begin upload segment. Dec 26 11:43:02.943481626 (500h14m2.814564671s) [noise=true , address=145.79.227.46:4603 , piece_id=QGECIFBCGTVK4BFAADEPB7AN4CEEV3YQEQAG7XVLX6PTUA6ZCYHA, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.943586476 (500h14m2.814669521s) [noise=true , address=45.91.116.12:4226 , piece_id=IJVZGCOZCQNJ7XPZ2X5ZAOBJ44UXAPC4EP5DBPILBWBTOGSQE2CA, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.944283653 (500h14m2.815366698s) [noise=true , address=77.90.31.19:4844 , piece_id=47PJR5G7ZUXBKUDSAYV2IEP44ILGVE4TTOPERLNOKI6II56SC3WQ, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.944746334 (500h14m2.815829379s) [noise=true , address=45.91.116.10:4084 , piece_id=A3F7N5VC7PKUHOPCMFJDLXRC66JWAJUSBVPHCKWXUD3DN3QS4U2A, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.945155062 (500h14m2.816238108s) [noise=true , address=77.90.31.14:4664 , piece_id=UV5FJNAFSA24B3UV7DD2FOIZBN6Q6SGJVUFOW6JL42JTLYCQ6QPA, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.945577507 (500h14m2.816660552s) [noise=true , address=145.79.227.43:4157 , piece_id=3AGZ7W3LG5BHYCCYYBKABOIHDIX62JKEOW622MNPR2X522GMQMTA, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.946013176 (500h14m2.817096221s) [noise=true , address=145.79.227.17:4971 , piece_id=64K7OBJEFCUXFQYHMFHAWQYYR36O74AQYSU7EFR463A6ZBBGYYDQ, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.946353334 (500h14m2.817436379s) [noise=true , address=45.91.116.8:4046 , piece_id=XZSJRP3GUVBX6TQA32S5VCF6ZZAY2SPCVOKFJS7GQWFCJMOWEV3A, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.946403359 (500h14m2.817486404s) [noise=true , address=45.91.116.5:4298 , piece_id=YQLNYJG6A5VRGEIKEVSUSWJ2SK5DDN53XOLHDPSC6IPVDIFMPUNA, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.946819962 (500h14m2.817903007s) [noise=true , address=145.79.227.52:4447 , piece_id=HRYRMA3VE23CSJ3FRDOBUYRVDA4NKP62GL5D3JK3463PUZF6LD3Q, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.947171472 (500h14m2.818254527s) [noise=true , address=145.79.227.34:4907 , piece_id=CMYXL6HPRP36RBKLYVLSDKRRDEBPIVXEDY4ME3PL5X2RS7ZCZM2Q, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.947194876 (500h14m2.818277921s) [noise=true , address=145.79.227.29:4943 , piece_id=PZCHZEHOE2WIWLMCXGLOUZAKIL77EREQBEAIMXTEFLY5Y3H74VYA, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.947677044 (500h14m2.818760089s) [noise=true , address=45.91.116.9:4264 , piece_id=TCT6LOQMSVHCGO66L4XHM6SHICJMUCUARVWJX2TODFFJMB2WXNGA, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.948225429 (500h14m2.819308544s) [noise=true , address=145.79.227.51:4409 , piece_id=H62IFOSSQTEBNTFODIP3DSSRYQXETW57WN7SK642L5NEH6BVX2VQ, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.948790925 (500h14m2.81987397s) [noise=true , address=45.91.116.6:4974 , piece_id=RLXG7YTNDUIKLKRZZULTEJR66VORCITLDTU4PNHVMEHGDWSRCXTA, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:02.949259729 (500h14m2.820342774s) [noise=true , address=45.91.116.13:4154 , piece_id=5O4LSWDMZIWHC7Q6SBIS76SYGKQL6WMOMZETCKANUT2EPBBGIADQ, seg_pos={0 0}, upload=439041]: Uploading piece... Dec 26 11:43:04.651618675 (500h14m4.52270172s) [upload=439041]: Next returned nil segment Dec 26 11:43:04.651644063 (500h14m4.522727118s) [upload=439041]: Waiting for error group managing segments... Dec 26 11:43:04.939927075 (500h14m4.81101011s) [noise=true , address=145.79.227.46:4603 , piece_id=QGECIFBCGTVK4BFAADEPB7AN4CEEV3YQEQAG7XVLX6PTUA6ZCYHA, seg_pos={0 0}, upload=439041]: Done uploading piece. err: <nil> Dec 26 11:43:04.958907327 (500h14m4.829990372s) [noise=true , address=145.79.227.52:4447 , piece_id=HRYRMA3VE23CSJ3FRDOBUYRVDA4NKP62GL5D3JK3463PUZF6LD3Q, seg_pos={0 0}, upload=439041]: Done uploading piece. err: <nil> Dec 26 11:43:04.959497801 (500h14m4.830580846s) [noise=true , address=145.79.227.29:4943 , piece_id=PZCHZEHOE2WIWLMCXGLOUZAKIL77EREQBEAIMXTEFLY5Y3H74VYA, seg_pos={0 0}, upload=439041]: Done uploading piece. err: <nil> Dec 26 11:43:04.970918169 (500h14m4.842001214s) [noise=true , address=145.79.227.17:4971 , piece_id=64K7OBJEFCUXFQYHMFHAWQYYR36O74AQYSU7EFR463A6ZBBGYYDQ, seg_pos={0 0}, upload=439041]: Done uploading piece. err: <nil> Dec 26 11:43:04.976226957 (500h14m4.847309992s) [noise=true , address=145.79.227.43:4157 , piece_id=3AGZ7W3LG5BHYCCYYBKABOIHDIX62JKEOW622MNPR2X522GMQMTA, seg_pos={0 0}, upload=439041]: Done uploading piece. err: <nil> Dec 26 11:43:04.979618925 (500h14m4.85070197s) [noise=true , address=145.79.227.51:4409 , piece_id=H62IFOSSQTEBNTFODIP3DSSRYQXETW57WN7SK642L5NEH6BVX2VQ, seg_pos={0 0}, upload=439041]: Done uploading piece. err: <nil> Dec 26 11:43:05.186340601 (500h14m5.057423646s) [noise=true , address=45.91.116.6:4974 , piece_id=RLXG7YTNDUIKLKRZZULTEJR66VORCITLDTU4PNHVMEHGDWSRCXTA, seg_pos={0 0}, upload=439041]: Done uploading piece. err: <nil> Dec 26 11:43:05.195561289 (500h14m5.066644344s) [noise=true , address=45.91.116.8:4046 , piece_id=XZSJRP3GUVBX6TQA32S5VCF6ZZAY2SPCVOKFJS7GQWFCJMOWEV3A, seg_pos={0 0}, upload=439041]: Done uploading piece. err: <nil> Dec 26 11:43:05.233599741 (500h14m5.104682786s) [noise=true , address=45.91.116.12:4226 , piece_id=IJVZGCOZCQNJ7XPZ2X5ZAOBJ44UXAPC4EP5DBPILBWBTOGSQE2CA, seg_pos={0 0}, upload=439041]: Done uploading piece. err: <nil> Dec 26 11:43:05.249595820 (500h14m5.120678865s) [noise=true , address=77.90.31.14:4664 , piece_id=UV5FJNAFSA24B3UV7DD2FOIZBN6Q6SGJVUFOW6JL42JTLYCQ6QPA, seg_pos={0 0}, upload=439041]: Done uploading piece. err: <nil> Dec 26 11:43:05.280396631 (500h14m5.151479676s) [noise=true , address=45.91.116.13:4154 , piece_id=5O4LSWDMZIWHC7Q6SBIS76SYGKQL6WMOMZETCKANUT2EPBBGIADQ, seg_pos={0 0}, upload=439041]: Done uploading piece. err: <nil> Dec 26 11:43:05.376228501 (500h14m5.247311536s) [noise=true , address=45.91.116.5:4298 , piece_id=YQLNYJG6A5VRGEIKEVSUSWJ2SK5DDN53XOLHDPSC6IPVDIFMPUNA, seg_pos={0 0}, upload=439041]: Done uploading piece. err: <nil> Dec 26 11:43:05.377105712 (500h14m5.248188747s) [noise=true , address=45.91.116.10:4084 , piece_id=A3F7N5VC7PKUHOPCMFJDLXRC66JWAJUSBVPHCKWXUD3DN3QS4U2A, seg_pos={0 0}, upload=439041]: Done uploading piece. err: <nil> Dec 26 11:43:05.377116212 (500h14m5.248199247s) [seg_pos={0 0}, upload=439041]: Segment reached optimal threshold of 13 pieces. Dec 26 11:43:05.377299871 (500h14m5.248382906s) [noise=true , address=145.79.227.34:4907 , piece_id=CMYXL6HPRP36RBKLYVLSDKRRDEBPIVXEDY4ME3PL5X2RS7ZCZM2Q, seg_pos={0 0}, upload=439041]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 11:43:05.377306514 (500h14m5.248389549s) [noise=true , address=145.79.227.34:4907 , piece_id=CMYXL6HPRP36RBKLYVLSDKRRDEBPIVXEDY4ME3PL5X2RS7ZCZM2Q, seg_pos={0 0}, upload=439041]: Long tail cancellation due to optimal threshold Dec 26 11:43:05.377570898 (500h14m5.248653933s) [noise=true , address=77.90.31.19:4844 , piece_id=47PJR5G7ZUXBKUDSAYV2IEP44ILGVE4TTOPERLNOKI6II56SC3WQ, seg_pos={0 0}, upload=439041]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: rpc: dial tcp 77.90.31.19:4844: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 11:43:05.377577220 (500h14m5.248660255s) [noise=true , address=77.90.31.19:4844 , piece_id=47PJR5G7ZUXBKUDSAYV2IEP44ILGVE4TTOPERLNOKI6II56SC3WQ, seg_pos={0 0}, upload=439041]: Long tail cancellation due to optimal threshold Dec 26 11:43:05.378051663 (500h14m5.249134708s) [noise=true , address=45.91.116.9:4264 , piece_id=TCT6LOQMSVHCGO66L4XHM6SHICJMUCUARVWJX2TODFFJMB2WXNGA, seg_pos={0 0}, upload=439041]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: read tcp 188.166.225.170:49576->45.91.116.9:4264: use of closed network connection; context canceled Dec 26 11:43:05.378072653 (500h14m5.249155698s) [noise=true , address=45.91.116.9:4264 , piece_id=TCT6LOQMSVHCGO66L4XHM6SHICJMUCUARVWJX2TODFFJMB2WXNGA, seg_pos={0 0}, upload=439041]: Long tail cancellation due to optimal threshold Dec 26 11:43:05.378109363 (500h14m5.249192408s) [seg_pos={0 0}, upload=439041]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:43:05.378219383 (500h14m5.249302418s) [upload=439041]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:43:05.378226055 (500h14m5.2493091s) [upload=439041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:43:05.673634416 (500h14m5.544717461s) [upload=439041]: Done uploading segments... Dec 26 11:43:05.673654775 (500h14m5.54473781s) [upload=439041]: Upload finished. err: <nil> Duration: 30.003775254s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................................................X....X] duration: 5.002111436s trace: https://jaeger.dev.storj.io/trace/10a24935a8f16015 FAIL (2025-12-26 11:43:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002135813s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.4777144s trace: https://jaeger.dev.storj.io/trace/3b628ab935afa966 FAIL (2025-12-26 11:42:50 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 6.477739888s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X..X.X......XX......XXX.XX......X.X..XX.X.X...X....X......] duration: 6.064073335s trace: https://jaeger.dev.storj.io/trace/730d54fb15e42c9 FAIL (2025-12-26 11:42:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: BUPXRO3RW37VVAD7STCKDV7MEL5IQ4XFGT7LV2TPL5GMYMR6HLNQ) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: SKLLCYFUPOQONKRHMVWQN2A7LTGYISC7U2EEICCUAHATLW4FHQ4A) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: RIJV3ILEDXMB4LIGMV4ZLPIIYGSV33JWESZMQECHMQIKLOAKYOBQ) context deadline exceeded ======LOGS====== Dec 26 11:36:56.505515043 (500h7m56.376598088s) [upload=438945]: Starting upload Dec 26 11:36:56.505618961 (500h7m56.376701996s) [upload=438945]: Done starting upload Dec 26 11:36:56.505837156 (500h7m56.376920212s) [upload=438945]: Uploading segments... Dec 26 11:36:56.506075150 (500h7m56.377158185s) [upload=438945]: Got next segment. Inline: false Dec 26 11:36:56.506117180 (500h7m56.377200215s) [upload=438945]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 11:36:56.506123451 (500h7m56.377206496s) [upload=438945]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 11:36:56.750379779 (500h7m56.621462814s) [seg_pos={0 0}, upload=438945]: Begin upload segment... Dec 26 11:36:56.750422250 (500h7m56.621505285s) [seg_pos={0 0}, upload=438945]: Done begin upload segment. Dec 26 11:36:56.750448380 (500h7m56.621531415s) [noise=true , address=145.79.227.4:4187 , piece_id=B7M6DVR5MO6LTCFI7XR3RY5UWO4JCUSHPSU54SM6OBX5OMUYUHOQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.750524154 (500h7m56.621607189s) [noise=true , address=145.79.227.29:4941 , piece_id=JEMZO23FYHC2XJNQIQM5N6L3Y755GAHYB3XEE4XAKR5QPDPGWAKQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.751138414 (500h7m56.622221449s) [noise=true , address=145.79.227.36:4519 , piece_id=DA2BGTCL7OSWGONK6SYEET2MUYWVYBAXQSYBQ63AYUX47QEVBH4A, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.751453734 (500h7m56.622536769s) [noise=true , address=77.90.31.14:4661 , piece_id=RIJV3ILEDXMB4LIGMV4ZLPIIYGSV33JWESZMQECHMQIKLOAKYOBQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.751751742 (500h7m56.622834777s) [noise=true , address=145.79.227.35:4481 , piece_id=ZRLEPLBXXZ2MXQCUF7RDJHHFOZK2DQ3ONBD6APNBZZR6O5HI3RNA, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.752042666 (500h7m56.623125701s) [noise=true , address=145.79.227.52:4445 , piece_id=2Z2EBGR6LDCPB6VLER4RYPVDRT6YZQCSJSFKSHSKRTUNHNKJ5QLA, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.752457315 (500h7m56.62354035s) [noise=true , address=45.91.116.13:4153 , piece_id=MNJ3SOQQ3DKQMHEKMOX6CTP2RGABGH265LCWJPTOUK7EEFZUYVAQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.752715697 (500h7m56.623798732s) [noise=true , address=145.79.227.44:4194 , piece_id=25EWP6UT3JJRC4KYRHFH562MMWL244DH67QF5PCGZSJ5DKSZQMPA, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.753043872 (500h7m56.624126907s) [noise=true , address=45.91.116.14:4192 , piece_id=BUPXRO3RW37VVAD7STCKDV7MEL5IQ4XFGT7LV2TPL5GMYMR6HLNQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.754630302 (500h7m56.625713338s) [noise=true , address=145.79.227.16:4910 , piece_id=RMOVMJY6ZBHXW5CSVAK4KVJPZS4ZSY7VVV5BTWPYXUGFX2L7XX7A, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.755866918 (500h7m56.626949953s) [noise=true , address=45.91.116.11:4118 , piece_id=SKLLCYFUPOQONKRHMVWQN2A7LTGYISC7U2EEICCUAHATLW4FHQ4A, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.757061353 (500h7m56.628144388s) [noise=true , address=45.91.116.5:4297 , piece_id=Y5YU32R6FWZPM5SROGRHG3NWE7GVO54MNULIUGDGELMSFC52P7QQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.758350327 (500h7m56.629433362s) [noise=true , address=145.79.227.46:4602 , piece_id=TEWX3BY3YSEH5GZA3ML35GJUN42NV3L2YKYWVI7724PJCLGNHO5Q, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.759633310 (500h7m56.630716345s) [noise=true , address=145.79.227.41:4085 , piece_id=WR46TBHJSSONTI6SEQLGNOPSU3SK2LFV4LFX6TYVQQRU7YYV6LPQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.760920681 (500h7m56.632003726s) [noise=true , address=45.91.116.8:4046 , piece_id=NGGRF6S3F2UMQ2GVIEY6EHAWKKYV75PHGHGIG24AISFEKAJDGSOQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:56.762417291 (500h7m56.633500336s) [noise=true , address=145.79.227.42:4121 , piece_id=EUQVW2HYMPLXFJR7MADFAKVSIWXC6ABCGNNM5FMCNWLZYDN6WLEQ, seg_pos={0 0}, upload=438945]: Uploading piece... Dec 26 11:36:58.481396575 (500h7m58.35247961s) [upload=438945]: Next returned nil segment Dec 26 11:36:58.481413537 (500h7m58.352496572s) [upload=438945]: Waiting for error group managing segments... Dec 26 11:36:58.762340483 (500h7m58.633423518s) [noise=true , address=145.79.227.36:4519 , piece_id=DA2BGTCL7OSWGONK6SYEET2MUYWVYBAXQSYBQ63AYUX47QEVBH4A, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.769558617 (500h7m58.640641663s) [noise=true , address=145.79.227.35:4481 , piece_id=ZRLEPLBXXZ2MXQCUF7RDJHHFOZK2DQ3ONBD6APNBZZR6O5HI3RNA, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.775582498 (500h7m58.646665533s) [noise=true , address=145.79.227.52:4445 , piece_id=2Z2EBGR6LDCPB6VLER4RYPVDRT6YZQCSJSFKSHSKRTUNHNKJ5QLA, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.780348644 (500h7m58.651431689s) [noise=true , address=145.79.227.4:4187 , piece_id=B7M6DVR5MO6LTCFI7XR3RY5UWO4JCUSHPSU54SM6OBX5OMUYUHOQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.785099239 (500h7m58.656182284s) [noise=true , address=145.79.227.29:4941 , piece_id=JEMZO23FYHC2XJNQIQM5N6L3Y755GAHYB3XEE4XAKR5QPDPGWAKQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.791439683 (500h7m58.662522718s) [noise=true , address=145.79.227.46:4602 , piece_id=TEWX3BY3YSEH5GZA3ML35GJUN42NV3L2YKYWVI7724PJCLGNHO5Q, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.792281556 (500h7m58.663364591s) [noise=true , address=145.79.227.16:4910 , piece_id=RMOVMJY6ZBHXW5CSVAK4KVJPZS4ZSY7VVV5BTWPYXUGFX2L7XX7A, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.794862450 (500h7m58.665945516s) [noise=true , address=145.79.227.42:4121 , piece_id=EUQVW2HYMPLXFJR7MADFAKVSIWXC6ABCGNNM5FMCNWLZYDN6WLEQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:58.795905156 (500h7m58.666988201s) [noise=true , address=145.79.227.41:4085 , piece_id=WR46TBHJSSONTI6SEQLGNOPSU3SK2LFV4LFX6TYVQQRU7YYV6LPQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:59.013511989 (500h7m58.884595024s) [noise=true , address=77.90.31.14:4661 , piece_id=RIJV3ILEDXMB4LIGMV4ZLPIIYGSV33JWESZMQECHMQIKLOAKYOBQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:59.101556480 (500h7m58.972639516s) [noise=true , address=45.91.116.8:4046 , piece_id=NGGRF6S3F2UMQ2GVIEY6EHAWKKYV75PHGHGIG24AISFEKAJDGSOQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:59.109052565 (500h7m58.9801356s) [noise=true , address=45.91.116.11:4118 , piece_id=SKLLCYFUPOQONKRHMVWQN2A7LTGYISC7U2EEICCUAHATLW4FHQ4A, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:59.126268548 (500h7m58.997351583s) [noise=true , address=45.91.116.14:4192 , piece_id=BUPXRO3RW37VVAD7STCKDV7MEL5IQ4XFGT7LV2TPL5GMYMR6HLNQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: <nil> Dec 26 11:36:59.126296330 (500h7m58.997379365s) [seg_pos={0 0}, upload=438945]: Segment reached optimal threshold of 13 pieces. Dec 26 11:36:59.126530205 (500h7m58.99761325s) [noise=true , address=45.91.116.13:4153 , piece_id=MNJ3SOQQ3DKQMHEKMOX6CTP2RGABGH265LCWJPTOUK7EEFZUYVAQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 26 11:36:59.126541096 (500h7m58.997624131s) [noise=true , address=45.91.116.13:4153 , piece_id=MNJ3SOQQ3DKQMHEKMOX6CTP2RGABGH265LCWJPTOUK7EEFZUYVAQ, seg_pos={0 0}, upload=438945]: Long tail cancellation due to optimal threshold Dec 26 11:36:59.126640455 (500h7m58.99772349s) [noise=true , address=145.79.227.44:4194 , piece_id=25EWP6UT3JJRC4KYRHFH562MMWL244DH67QF5PCGZSJ5DKSZQMPA, seg_pos={0 0}, upload=438945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 11:36:59.126695530 (500h7m58.997778575s) [noise=true , address=145.79.227.44:4194 , piece_id=25EWP6UT3JJRC4KYRHFH562MMWL244DH67QF5PCGZSJ5DKSZQMPA, seg_pos={0 0}, upload=438945]: Long tail cancellation due to optimal threshold Dec 26 11:36:59.126897886 (500h7m58.997980921s) [noise=true , address=45.91.116.5:4297 , piece_id=Y5YU32R6FWZPM5SROGRHG3NWE7GVO54MNULIUGDGELMSFC52P7QQ, seg_pos={0 0}, upload=438945]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 11:36:59.126908406 (500h7m58.997991441s) [noise=true , address=45.91.116.5:4297 , piece_id=Y5YU32R6FWZPM5SROGRHG3NWE7GVO54MNULIUGDGELMSFC52P7QQ, seg_pos={0 0}, upload=438945]: Long tail cancellation due to optimal threshold Dec 26 11:36:59.126957128 (500h7m58.998040173s) [seg_pos={0 0}, upload=438945]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 11:36:59.127105982 (500h7m58.998189027s) [upload=438945]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 11:36:59.127115209 (500h7m58.998198255s) [upload=438945]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 11:36:59.423053990 (500h7m59.294137035s) [upload=438945]: Done uploading segments... Dec 26 11:36:59.423075740 (500h7m59.294158775s) [upload=438945]: Upload finished. err: <nil> Duration: 6.064096759s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)