Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [XXXXXX.XXXXXX.XX.XXXXXX.XXXX.XXXXXXXXXXXX..XXXXXXX.....XXX..] duration: 5.001523281s trace: https://jaeger.dev.storj.io/trace/165897763121c0e FAIL (2025-12-27 14:48:57 UTC) orbiter warning: context deadline exceeded Duration: 5.001556264s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................X.............X..X...........X.........] duration: 49.616222462s trace: https://jaeger.dev.storj.io/trace/4d753dda46642d0d FAIL (2025-12-27 14:48:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd, Piece ID: 5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA) context deadline exceeded; piecestore: (Node ID: 12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ, Piece ID: ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ) context deadline exceeded; piecestore: (Node ID: 12cth5bdi5iAWhHcLxqN9c7GLdheS7SLQ3x5fEHG1MdgdXWFdD6, Piece ID: OCZIJHDS4AESHNCHUJIBVN47X336BIKXBLIQ4BVYSF7LAPWIS3KA) context deadline exceeded ======LOGS====== 660 (527h10m58.618722705s) [noise=true , address=77.90.31.19:4860 , piece_id=5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748001970 (527h10m58.619085005s) [noise=true , address=145.79.227.44:4205 , piece_id=HVNDI6C4SQZYPZIEEP7TC5RW6YNW5WGFDOJKQIIAN3UH3VGDXNIA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748340274 (527h10m58.619423319s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748682386 (527h10m58.619765421s) [noise=true , address=77.90.31.10:4530 , piece_id=K22R6V3CWZWVFKROB23NKYFC5R2Z5LT4KSXMTXFQUUROJKCJWESA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748994651 (527h10m58.620077696s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.749319318 (527h10m58.620402363s) [noise=true , address=145.79.227.40:4060 , piece_id=OADO7RXPLZ4NLGUC2CCXNJQ6OEMU5JWTXCRY7WFWYKKGLMVU6YBA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.749662011 (527h10m58.620745046s) [noise=true , address=77.90.31.12:4601 , piece_id=LCBL5PH44YSNTX6W3UBMGJAGSOTBB6CKMBBD7E3GKDFQBUXMR3AA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.750250261 (527h10m58.621333296s) [noise=true , address=145.79.227.46:4633 , piece_id=U6GO7WKUMICS7QYSJZXZ6ZB33S2MYCV5JGEPVHEG7BHX5HGZL3XA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.750568958 (527h10m58.621651993s) [noise=true , address=145.79.227.23:4579 , piece_id=A3LOK5DMHEU3G3C6XYC2CWLOL7U4XJFXK4RV4A3PH4AXRHTA4EGQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.751689872 (527h10m58.622772917s) [noise=true , address=45.91.116.8:4049 , piece_id=ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.751758763 (527h10m58.622841809s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.753063619 (527h10m58.624146654s) [noise=true , address=145.79.227.36:4537 , piece_id=TMCDI7JYHU6MAPAGTLJTB2GSXMSBPJZJSPZWGAFF5I4VGEGNOMUQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.753568099 (527h10m58.624651134s) [noise=true , address=145.79.227.8:4337 , piece_id=BC5P5Y75VGZDTNDWEOC27UTK3TMWFRT7BNKQTJAYYHPYLYRXV2VQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.754895386 (527h10m58.625978421s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.755999769 (527h10m58.627082804s) [noise=true , address=77.90.31.21:4893 , piece_id=WYF4STSQ27E7KEMRIXKC4ILRHKN4UCNGH5NJFZ5SHOBKW3FNUCHA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.757486991 (527h10m58.628570037s) [noise=true , address=64.49.13.154:4933 , piece_id=V7STA7QTESDS5FDALZIPURQRCO6X4MONA2PTU5RIJHM34P5BQTNA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:40:00.120776880 (527h10m59.991859915s) [upload=462397]: Next returned nil segment Dec 27 14:40:00.120795044 (527h10m59.991878079s) [upload=462397]: Waiting for error group managing segments... Dec 27 14:40:00.475194542 (527h11m0.346277587s) [noise=true , address=64.49.13.154:4933 , piece_id=V7STA7QTESDS5FDALZIPURQRCO6X4MONA2PTU5RIJHM34P5BQTNA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.626897129 (527h11m0.497980174s) [noise=true , address=145.79.227.46:4633 , piece_id=U6GO7WKUMICS7QYSJZXZ6ZB33S2MYCV5JGEPVHEG7BHX5HGZL3XA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.634206858 (527h11m0.505289903s) [noise=true , address=145.79.227.34:4927 , piece_id=3PJB2HW34B5N5LADQYZ3FPNEIWBGKKGJEGADMPJIMWRG6HGQG36Q, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.636936777 (527h11m0.508019822s) [noise=true , address=145.79.227.36:4537 , piece_id=TMCDI7JYHU6MAPAGTLJTB2GSXMSBPJZJSPZWGAFF5I4VGEGNOMUQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.650341221 (527h11m0.521424266s) [noise=true , address=145.79.227.8:4337 , piece_id=BC5P5Y75VGZDTNDWEOC27UTK3TMWFRT7BNKQTJAYYHPYLYRXV2VQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.662509752 (527h11m0.533592797s) [noise=true , address=145.79.227.23:4579 , piece_id=A3LOK5DMHEU3G3C6XYC2CWLOL7U4XJFXK4RV4A3PH4AXRHTA4EGQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.820994414 (527h11m0.692077449s) [noise=true , address=77.90.31.12:4601 , piece_id=LCBL5PH44YSNTX6W3UBMGJAGSOTBB6CKMBBD7E3GKDFQBUXMR3AA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.908728394 (527h11m0.779811429s) [noise=true , address=45.91.116.8:4049 , piece_id=ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.921937597 (527h11m0.793020642s) [noise=true , address=77.90.31.10:4530 , piece_id=K22R6V3CWZWVFKROB23NKYFC5R2Z5LT4KSXMTXFQUUROJKCJWESA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.180318328 (527h11m1.051401374s) [noise=true , address=77.90.31.14:4683 , piece_id=OCZIJHDS4AESHNCHUJIBVN47X336BIKXBLIQ4BVYSF7LAPWIS3KA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.467385073 (527h11m1.338468118s) [noise=true , address=77.90.31.19:4860 , piece_id=5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.506047844 (527h11m1.377130919s) [noise=true , address=145.79.227.44:4205 , piece_id=HVNDI6C4SQZYPZIEEP7TC5RW6YNW5WGFDOJKQIIAN3UH3VGDXNIA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.551036449 (527h11m1.422119494s) [noise=true , address=145.79.227.40:4060 , piece_id=OADO7RXPLZ4NLGUC2CCXNJQ6OEMU5JWTXCRY7WFWYKKGLMVU6YBA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.567791785 (527h11m1.43887482s) [noise=true , address=77.90.31.21:4893 , piece_id=WYF4STSQ27E7KEMRIXKC4ILRHKN4UCNGH5NJFZ5SHOBKW3FNUCHA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.567835187 (527h11m1.438918232s) [seg_pos={0 0}, upload=462397]: Segment reached optimal threshold of 14 pieces. Dec 27 14:40:01.568097526 (527h11m1.439180581s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568109259 (527h11m1.439192304s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568506055 (527h11m1.4395891s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568516865 (527h11m1.43959991s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568673093 (527h11m1.439756148s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection Dec 27 14:40:01.568680948 (527h11m1.439763993s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568844680 (527h11m1.439927725s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FtB4bKTHntdUQxF8G5o9nT73oS2xB7fgD44BkRfKDNFyZnKCk): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568852474 (527h11m1.439935519s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568872633 (527h11m1.439955678s) [seg_pos={0 0}, upload=462397]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 14:40:01.568960319 (527h11m1.440043365s) [upload=462397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:40:01.568966050 (527h11m1.440049095s) [upload=462397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:40:01.862585703 (527h11m1.733668748s) [upload=462397]: Done uploading segments... Dec 27 14:40:01.862613346 (527h11m1.733696401s) [upload=462397]: Upload finished. err: <nil> Duration: 49.61624802s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X..X...XX.............X..X....X.........X.X....X...X.....X..] duration: 7.273859811s trace: https://jaeger.dev.storj.io/trace/38b797e462beb605 FAIL (2025-12-27 14:48:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: MGASCARKZ6424UA3HAJUOI6QC2Z4H5PTG52YLUFM47NR4JSDMBKQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: U2DLXXCX2NEHH7FMRROW3WB74RO7BNOVCJE55OHHBDSJESFQQWAQ) context deadline exceeded ======LOGS====== Dec 27 14:47:33.790690098 (527h18m33.661773143s) [upload=462504]: Starting upload Dec 27 14:47:33.790752285 (527h18m33.66183533s) [upload=462504]: Done starting upload Dec 27 14:47:33.790909084 (527h18m33.661992139s) [upload=462504]: Uploading segments... Dec 27 14:47:33.791052798 (527h18m33.662135863s) [upload=462504]: Got next segment. Inline: false Dec 27 14:47:33.791111610 (527h18m33.662194645s) [upload=462504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 14:47:33.791117862 (527h18m33.662200907s) [upload=462504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 14:47:34.036515711 (527h18m33.907598766s) [seg_pos={0 0}, upload=462504]: Begin upload segment... Dec 27 14:47:34.036577168 (527h18m33.907660214s) [seg_pos={0 0}, upload=462504]: Done begin upload segment. Dec 27 14:47:34.036602928 (527h18m33.907685973s) [noise=true , address=145.79.227.50:4374 , piece_id=PQBTCCNOFDPJ7UTUPGOXC42NOL5ZRPPLMVMTZR2O6YPZQLBT4PGA, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.036621463 (527h18m33.907704508s) [noise=true , address=77.90.31.17:4773 , piece_id=OUX67FKMVDHYXXUOGBRS26Z3RRQOQGZ75SER7LPUFW27DXU6GSUA, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.037190817 (527h18m33.908273862s) [noise=true , address=77.90.31.10:4519 , piece_id=BF564TG2HBB2USZEFRRO2OFHVOD4QNKHQISSDUCWFW55LRCKXK7Q, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.038468861 (527h18m33.909551896s) [noise=true , address=145.79.227.41:4085 , piece_id=QBXCN372TIX2K3OOJ3R4IDEKGCPZJGJQWEGEQ3ISKZPE7OPSNWSA, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.039027585 (527h18m33.91011063s) [noise=true , address=77.90.31.15:4700 , piece_id=INWYR4O6ILASY5TLXUGH7AAJ7Z5BRFPTVWKWQTGGRD2OQPFQIFFA, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.039550631 (527h18m33.910633676s) [noise=true , address=45.91.116.10:4081 , piece_id=MFZ3F2O3KRQ7SAQWGNHVNLWPQUBFOVC75CZNZMTKDSHDC3ZBDCWA, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.041646993 (527h18m33.912730038s) [noise=true , address=45.91.116.7:4009 , piece_id=SFSUC6FHT5CNQR4J3RYK36APS4G4AMBVXYEBKDX5XNRWXBASO5SQ, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.043650848 (527h18m33.914733923s) [noise=true , address=77.90.31.18:4805 , piece_id=Y7FVDVV3IEDY4XDFI67XZAL7D2SFVKQSD47CYMJV7GWY4XRGUNFQ, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.045644835 (527h18m33.91672788s) [noise=true , address=45.91.116.9:4263 , piece_id=U2DLXXCX2NEHH7FMRROW3WB74RO7BNOVCJE55OHHBDSJESFQQWAQ, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.046817898 (527h18m33.917900944s) [noise=true , address=145.79.227.49:4338 , piece_id=JHGXIUN2XR6N7IUAXJRYKTKY3SHA4NKPM22RQK5UI2WUXTGT4RQQ, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.048385483 (527h18m33.919468528s) [noise=true , address=45.91.116.5:4297 , piece_id=SDMA6DBBTHSY2XRTGAVVLKBGVPITNYKNDVXWWB5WJTRQQHLO5SCQ, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.048945000 (527h18m33.920028045s) [noise=true , address=145.79.227.37:4567 , piece_id=QOACCYWCR5KLJMVZPTIXYMN7LZD6FHD5UDPF2CT3WLRKH7EYYKBQ, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.050550727 (527h18m33.921633772s) [noise=true , address=77.90.31.11:4554 , piece_id=D5VC6SJIH4ISSJ35UWA5NTE7QZTV6R4ZP5PY3LDRMRFBNDGV4LCQ, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.051751203 (527h18m33.922834248s) [noise=true , address=145.79.227.39:4014 , piece_id=FVJSRVM565PUJN4HZLBOFPI3M5EWDOFYY2UUUSTZSWZKN6USCULA, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.052569822 (527h18m33.923652857s) [noise=true , address=45.91.116.12:4226 , piece_id=2QIM5YKXSEYLEJEFAWDJGL75YGHAE5XPCS4RBDPLMPTRP3L4E53A, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.054092180 (527h18m33.925175225s) [noise=true , address=45.91.116.6:4973 , piece_id=MGASCARKZ6424UA3HAJUOI6QC2Z4H5PTG52YLUFM47NR4JSDMBKQ, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:35.741527659 (527h18m35.612610694s) [upload=462504]: Next returned nil segment Dec 27 14:47:35.741541395 (527h18m35.61262444s) [upload=462504]: Waiting for error group managing segments... Dec 27 14:47:36.028778814 (527h18m35.899861859s) [noise=true , address=145.79.227.50:4374 , piece_id=PQBTCCNOFDPJ7UTUPGOXC42NOL5ZRPPLMVMTZR2O6YPZQLBT4PGA, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.055964650 (527h18m35.927047695s) [noise=true , address=145.79.227.37:4567 , piece_id=QOACCYWCR5KLJMVZPTIXYMN7LZD6FHD5UDPF2CT3WLRKH7EYYKBQ, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.056431579 (527h18m35.927514614s) [noise=true , address=145.79.227.39:4014 , piece_id=FVJSRVM565PUJN4HZLBOFPI3M5EWDOFYY2UUUSTZSWZKN6USCULA, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.076200563 (527h18m35.947283608s) [noise=true , address=145.79.227.41:4085 , piece_id=QBXCN372TIX2K3OOJ3R4IDEKGCPZJGJQWEGEQ3ISKZPE7OPSNWSA, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.078852502 (527h18m35.949935547s) [noise=true , address=145.79.227.49:4338 , piece_id=JHGXIUN2XR6N7IUAXJRYKTKY3SHA4NKPM22RQK5UI2WUXTGT4RQQ, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.278837388 (527h18m36.149920433s) [noise=true , address=45.91.116.5:4297 , piece_id=SDMA6DBBTHSY2XRTGAVVLKBGVPITNYKNDVXWWB5WJTRQQHLO5SCQ, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.287310383 (527h18m36.158393428s) [noise=true , address=45.91.116.9:4263 , piece_id=U2DLXXCX2NEHH7FMRROW3WB74RO7BNOVCJE55OHHBDSJESFQQWAQ, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.333665580 (527h18m36.204748616s) [noise=true , address=45.91.116.6:4973 , piece_id=MGASCARKZ6424UA3HAJUOI6QC2Z4H5PTG52YLUFM47NR4JSDMBKQ, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.348001368 (527h18m36.219084414s) [noise=true , address=77.90.31.11:4554 , piece_id=D5VC6SJIH4ISSJ35UWA5NTE7QZTV6R4ZP5PY3LDRMRFBNDGV4LCQ, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.362907472 (527h18m36.233990508s) [noise=true , address=77.90.31.18:4805 , piece_id=Y7FVDVV3IEDY4XDFI67XZAL7D2SFVKQSD47CYMJV7GWY4XRGUNFQ, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.484591630 (527h18m36.355674675s) [noise=true , address=77.90.31.15:4700 , piece_id=INWYR4O6ILASY5TLXUGH7AAJ7Z5BRFPTVWKWQTGGRD2OQPFQIFFA, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.526615280 (527h18m36.397698325s) [noise=true , address=45.91.116.10:4081 , piece_id=MFZ3F2O3KRQ7SAQWGNHVNLWPQUBFOVC75CZNZMTKDSHDC3ZBDCWA, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.655545496 (527h18m36.526628531s) [noise=true , address=77.90.31.17:4773 , piece_id=OUX67FKMVDHYXXUOGBRS26Z3RRQOQGZ75SER7LPUFW27DXU6GSUA, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.655565775 (527h18m36.52664882s) [seg_pos={0 0}, upload=462504]: Segment reached optimal threshold of 13 pieces. Dec 27 14:47:36.655737682 (527h18m36.526820717s) [noise=true , address=77.90.31.10:4519 , piece_id=BF564TG2HBB2USZEFRRO2OFHVOD4QNKHQISSDUCWFW55LRCKXK7Q, seg_pos={0 0}, upload=462504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 27 14:47:36.655745828 (527h18m36.526828863s) [noise=true , address=77.90.31.10:4519 , piece_id=BF564TG2HBB2USZEFRRO2OFHVOD4QNKHQISSDUCWFW55LRCKXK7Q, seg_pos={0 0}, upload=462504]: Long tail cancellation due to optimal threshold Dec 27 14:47:36.655826531 (527h18m36.526909596s) [noise=true , address=45.91.116.7:4009 , piece_id=SFSUC6FHT5CNQR4J3RYK36APS4G4AMBVXYEBKDX5XNRWXBASO5SQ, seg_pos={0 0}, upload=462504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 27 14:47:36.655836960 (527h18m36.526919996s) [noise=true , address=45.91.116.7:4009 , piece_id=SFSUC6FHT5CNQR4J3RYK36APS4G4AMBVXYEBKDX5XNRWXBASO5SQ, seg_pos={0 0}, upload=462504]: Long tail cancellation due to optimal threshold Dec 27 14:47:36.656007666 (527h18m36.527090711s) [noise=true , address=45.91.116.12:4226 , piece_id=2QIM5YKXSEYLEJEFAWDJGL75YGHAE5XPCS4RBDPLMPTRP3L4E53A, seg_pos={0 0}, upload=462504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: read tcp 188.166.225.170:58942->45.91.116.12:4226: use of closed network connection; context canceled Dec 27 14:47:36.656013687 (527h18m36.527096722s) [noise=true , address=45.91.116.12:4226 , piece_id=2QIM5YKXSEYLEJEFAWDJGL75YGHAE5XPCS4RBDPLMPTRP3L4E53A, seg_pos={0 0}, upload=462504]: Long tail cancellation due to optimal threshold Dec 27 14:47:36.656028556 (527h18m36.527111591s) [seg_pos={0 0}, upload=462504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 14:47:36.656126823 (527h18m36.527209858s) [upload=462504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:47:36.656131532 (527h18m36.527214567s) [upload=462504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:47:36.954703433 (527h18m36.825786478s) [upload=462504]: Done uploading segments... Dec 27 14:47:36.954724975 (527h18m36.82580801s) [upload=462504]: Upload finished. err: <nil> Duration: 7.273883207s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 8.312954807s trace: https://jaeger.dev.storj.io/trace/2f00e104e333c43f FAIL (2025-12-27 14:48:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== }, upload=462172]: Uploading piece... Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments... Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces. Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments... Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil> Duration: 8.312981908s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [XX..X...XX.............X..X....X.........X.X....X...X.....X.] duration: 6.118073556s trace: https://jaeger.dev.storj.io/trace/38b797e462beb605 FAIL (2025-12-27 14:48:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: MGASCARKZ6424UA3HAJUOI6QC2Z4H5PTG52YLUFM47NR4JSDMBKQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: U2DLXXCX2NEHH7FMRROW3WB74RO7BNOVCJE55OHHBDSJESFQQWAQ) context deadline exceeded ======LOGS====== Dec 27 14:47:33.790690098 (527h18m33.661773143s) [upload=462504]: Starting upload Dec 27 14:47:33.790752285 (527h18m33.66183533s) [upload=462504]: Done starting upload Dec 27 14:47:33.790909084 (527h18m33.661992139s) [upload=462504]: Uploading segments... Dec 27 14:47:33.791052798 (527h18m33.662135863s) [upload=462504]: Got next segment. Inline: false Dec 27 14:47:33.791111610 (527h18m33.662194645s) [upload=462504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 14:47:33.791117862 (527h18m33.662200907s) [upload=462504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 14:47:34.036515711 (527h18m33.907598766s) [seg_pos={0 0}, upload=462504]: Begin upload segment... Dec 27 14:47:34.036577168 (527h18m33.907660214s) [seg_pos={0 0}, upload=462504]: Done begin upload segment. Dec 27 14:47:34.036602928 (527h18m33.907685973s) [noise=true , address=145.79.227.50:4374 , piece_id=PQBTCCNOFDPJ7UTUPGOXC42NOL5ZRPPLMVMTZR2O6YPZQLBT4PGA, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.036621463 (527h18m33.907704508s) [noise=true , address=77.90.31.17:4773 , piece_id=OUX67FKMVDHYXXUOGBRS26Z3RRQOQGZ75SER7LPUFW27DXU6GSUA, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.037190817 (527h18m33.908273862s) [noise=true , address=77.90.31.10:4519 , piece_id=BF564TG2HBB2USZEFRRO2OFHVOD4QNKHQISSDUCWFW55LRCKXK7Q, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.038468861 (527h18m33.909551896s) [noise=true , address=145.79.227.41:4085 , piece_id=QBXCN372TIX2K3OOJ3R4IDEKGCPZJGJQWEGEQ3ISKZPE7OPSNWSA, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.039027585 (527h18m33.91011063s) [noise=true , address=77.90.31.15:4700 , piece_id=INWYR4O6ILASY5TLXUGH7AAJ7Z5BRFPTVWKWQTGGRD2OQPFQIFFA, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.039550631 (527h18m33.910633676s) [noise=true , address=45.91.116.10:4081 , piece_id=MFZ3F2O3KRQ7SAQWGNHVNLWPQUBFOVC75CZNZMTKDSHDC3ZBDCWA, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.041646993 (527h18m33.912730038s) [noise=true , address=45.91.116.7:4009 , piece_id=SFSUC6FHT5CNQR4J3RYK36APS4G4AMBVXYEBKDX5XNRWXBASO5SQ, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.043650848 (527h18m33.914733923s) [noise=true , address=77.90.31.18:4805 , piece_id=Y7FVDVV3IEDY4XDFI67XZAL7D2SFVKQSD47CYMJV7GWY4XRGUNFQ, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.045644835 (527h18m33.91672788s) [noise=true , address=45.91.116.9:4263 , piece_id=U2DLXXCX2NEHH7FMRROW3WB74RO7BNOVCJE55OHHBDSJESFQQWAQ, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.046817898 (527h18m33.917900944s) [noise=true , address=145.79.227.49:4338 , piece_id=JHGXIUN2XR6N7IUAXJRYKTKY3SHA4NKPM22RQK5UI2WUXTGT4RQQ, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.048385483 (527h18m33.919468528s) [noise=true , address=45.91.116.5:4297 , piece_id=SDMA6DBBTHSY2XRTGAVVLKBGVPITNYKNDVXWWB5WJTRQQHLO5SCQ, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.048945000 (527h18m33.920028045s) [noise=true , address=145.79.227.37:4567 , piece_id=QOACCYWCR5KLJMVZPTIXYMN7LZD6FHD5UDPF2CT3WLRKH7EYYKBQ, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.050550727 (527h18m33.921633772s) [noise=true , address=77.90.31.11:4554 , piece_id=D5VC6SJIH4ISSJ35UWA5NTE7QZTV6R4ZP5PY3LDRMRFBNDGV4LCQ, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.051751203 (527h18m33.922834248s) [noise=true , address=145.79.227.39:4014 , piece_id=FVJSRVM565PUJN4HZLBOFPI3M5EWDOFYY2UUUSTZSWZKN6USCULA, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.052569822 (527h18m33.923652857s) [noise=true , address=45.91.116.12:4226 , piece_id=2QIM5YKXSEYLEJEFAWDJGL75YGHAE5XPCS4RBDPLMPTRP3L4E53A, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.054092180 (527h18m33.925175225s) [noise=true , address=45.91.116.6:4973 , piece_id=MGASCARKZ6424UA3HAJUOI6QC2Z4H5PTG52YLUFM47NR4JSDMBKQ, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:35.741527659 (527h18m35.612610694s) [upload=462504]: Next returned nil segment Dec 27 14:47:35.741541395 (527h18m35.61262444s) [upload=462504]: Waiting for error group managing segments... Dec 27 14:47:36.028778814 (527h18m35.899861859s) [noise=true , address=145.79.227.50:4374 , piece_id=PQBTCCNOFDPJ7UTUPGOXC42NOL5ZRPPLMVMTZR2O6YPZQLBT4PGA, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.055964650 (527h18m35.927047695s) [noise=true , address=145.79.227.37:4567 , piece_id=QOACCYWCR5KLJMVZPTIXYMN7LZD6FHD5UDPF2CT3WLRKH7EYYKBQ, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.056431579 (527h18m35.927514614s) [noise=true , address=145.79.227.39:4014 , piece_id=FVJSRVM565PUJN4HZLBOFPI3M5EWDOFYY2UUUSTZSWZKN6USCULA, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.076200563 (527h18m35.947283608s) [noise=true , address=145.79.227.41:4085 , piece_id=QBXCN372TIX2K3OOJ3R4IDEKGCPZJGJQWEGEQ3ISKZPE7OPSNWSA, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.078852502 (527h18m35.949935547s) [noise=true , address=145.79.227.49:4338 , piece_id=JHGXIUN2XR6N7IUAXJRYKTKY3SHA4NKPM22RQK5UI2WUXTGT4RQQ, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.278837388 (527h18m36.149920433s) [noise=true , address=45.91.116.5:4297 , piece_id=SDMA6DBBTHSY2XRTGAVVLKBGVPITNYKNDVXWWB5WJTRQQHLO5SCQ, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.287310383 (527h18m36.158393428s) [noise=true , address=45.91.116.9:4263 , piece_id=U2DLXXCX2NEHH7FMRROW3WB74RO7BNOVCJE55OHHBDSJESFQQWAQ, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.333665580 (527h18m36.204748616s) [noise=true , address=45.91.116.6:4973 , piece_id=MGASCARKZ6424UA3HAJUOI6QC2Z4H5PTG52YLUFM47NR4JSDMBKQ, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.348001368 (527h18m36.219084414s) [noise=true , address=77.90.31.11:4554 , piece_id=D5VC6SJIH4ISSJ35UWA5NTE7QZTV6R4ZP5PY3LDRMRFBNDGV4LCQ, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.362907472 (527h18m36.233990508s) [noise=true , address=77.90.31.18:4805 , piece_id=Y7FVDVV3IEDY4XDFI67XZAL7D2SFVKQSD47CYMJV7GWY4XRGUNFQ, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.484591630 (527h18m36.355674675s) [noise=true , address=77.90.31.15:4700 , piece_id=INWYR4O6ILASY5TLXUGH7AAJ7Z5BRFPTVWKWQTGGRD2OQPFQIFFA, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.526615280 (527h18m36.397698325s) [noise=true , address=45.91.116.10:4081 , piece_id=MFZ3F2O3KRQ7SAQWGNHVNLWPQUBFOVC75CZNZMTKDSHDC3ZBDCWA, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.655545496 (527h18m36.526628531s) [noise=true , address=77.90.31.17:4773 , piece_id=OUX67FKMVDHYXXUOGBRS26Z3RRQOQGZ75SER7LPUFW27DXU6GSUA, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.655565775 (527h18m36.52664882s) [seg_pos={0 0}, upload=462504]: Segment reached optimal threshold of 13 pieces. Dec 27 14:47:36.655737682 (527h18m36.526820717s) [noise=true , address=77.90.31.10:4519 , piece_id=BF564TG2HBB2USZEFRRO2OFHVOD4QNKHQISSDUCWFW55LRCKXK7Q, seg_pos={0 0}, upload=462504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 27 14:47:36.655745828 (527h18m36.526828863s) [noise=true , address=77.90.31.10:4519 , piece_id=BF564TG2HBB2USZEFRRO2OFHVOD4QNKHQISSDUCWFW55LRCKXK7Q, seg_pos={0 0}, upload=462504]: Long tail cancellation due to optimal threshold Dec 27 14:47:36.655826531 (527h18m36.526909596s) [noise=true , address=45.91.116.7:4009 , piece_id=SFSUC6FHT5CNQR4J3RYK36APS4G4AMBVXYEBKDX5XNRWXBASO5SQ, seg_pos={0 0}, upload=462504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 27 14:47:36.655836960 (527h18m36.526919996s) [noise=true , address=45.91.116.7:4009 , piece_id=SFSUC6FHT5CNQR4J3RYK36APS4G4AMBVXYEBKDX5XNRWXBASO5SQ, seg_pos={0 0}, upload=462504]: Long tail cancellation due to optimal threshold Dec 27 14:47:36.656007666 (527h18m36.527090711s) [noise=true , address=45.91.116.12:4226 , piece_id=2QIM5YKXSEYLEJEFAWDJGL75YGHAE5XPCS4RBDPLMPTRP3L4E53A, seg_pos={0 0}, upload=462504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: read tcp 188.166.225.170:58942->45.91.116.12:4226: use of closed network connection; context canceled Dec 27 14:47:36.656013687 (527h18m36.527096722s) [noise=true , address=45.91.116.12:4226 , piece_id=2QIM5YKXSEYLEJEFAWDJGL75YGHAE5XPCS4RBDPLMPTRP3L4E53A, seg_pos={0 0}, upload=462504]: Long tail cancellation due to optimal threshold Dec 27 14:47:36.656028556 (527h18m36.527111591s) [seg_pos={0 0}, upload=462504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 14:47:36.656126823 (527h18m36.527209858s) [upload=462504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:47:36.656131532 (527h18m36.527214567s) [upload=462504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:47:36.954703433 (527h18m36.825786478s) [upload=462504]: Done uploading segments... Dec 27 14:47:36.954724975 (527h18m36.82580801s) [upload=462504]: Upload finished. err: <nil> Duration: 6.118109595s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [XXX..X...XX.............X..X....X.........X.X....X...X.....X] duration: 30.003895954s trace: https://jaeger.dev.storj.io/trace/38b797e462beb605 FAIL (2025-12-27 14:48:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: MGASCARKZ6424UA3HAJUOI6QC2Z4H5PTG52YLUFM47NR4JSDMBKQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: U2DLXXCX2NEHH7FMRROW3WB74RO7BNOVCJE55OHHBDSJESFQQWAQ) context deadline exceeded ======LOGS====== Dec 27 14:47:33.790690098 (527h18m33.661773143s) [upload=462504]: Starting upload Dec 27 14:47:33.790752285 (527h18m33.66183533s) [upload=462504]: Done starting upload Dec 27 14:47:33.790909084 (527h18m33.661992139s) [upload=462504]: Uploading segments... Dec 27 14:47:33.791052798 (527h18m33.662135863s) [upload=462504]: Got next segment. Inline: false Dec 27 14:47:33.791111610 (527h18m33.662194645s) [upload=462504]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 14:47:33.791117862 (527h18m33.662200907s) [upload=462504]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 14:47:34.036515711 (527h18m33.907598766s) [seg_pos={0 0}, upload=462504]: Begin upload segment... Dec 27 14:47:34.036577168 (527h18m33.907660214s) [seg_pos={0 0}, upload=462504]: Done begin upload segment. Dec 27 14:47:34.036602928 (527h18m33.907685973s) [noise=true , address=145.79.227.50:4374 , piece_id=PQBTCCNOFDPJ7UTUPGOXC42NOL5ZRPPLMVMTZR2O6YPZQLBT4PGA, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.036621463 (527h18m33.907704508s) [noise=true , address=77.90.31.17:4773 , piece_id=OUX67FKMVDHYXXUOGBRS26Z3RRQOQGZ75SER7LPUFW27DXU6GSUA, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.037190817 (527h18m33.908273862s) [noise=true , address=77.90.31.10:4519 , piece_id=BF564TG2HBB2USZEFRRO2OFHVOD4QNKHQISSDUCWFW55LRCKXK7Q, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.038468861 (527h18m33.909551896s) [noise=true , address=145.79.227.41:4085 , piece_id=QBXCN372TIX2K3OOJ3R4IDEKGCPZJGJQWEGEQ3ISKZPE7OPSNWSA, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.039027585 (527h18m33.91011063s) [noise=true , address=77.90.31.15:4700 , piece_id=INWYR4O6ILASY5TLXUGH7AAJ7Z5BRFPTVWKWQTGGRD2OQPFQIFFA, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.039550631 (527h18m33.910633676s) [noise=true , address=45.91.116.10:4081 , piece_id=MFZ3F2O3KRQ7SAQWGNHVNLWPQUBFOVC75CZNZMTKDSHDC3ZBDCWA, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.041646993 (527h18m33.912730038s) [noise=true , address=45.91.116.7:4009 , piece_id=SFSUC6FHT5CNQR4J3RYK36APS4G4AMBVXYEBKDX5XNRWXBASO5SQ, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.043650848 (527h18m33.914733923s) [noise=true , address=77.90.31.18:4805 , piece_id=Y7FVDVV3IEDY4XDFI67XZAL7D2SFVKQSD47CYMJV7GWY4XRGUNFQ, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.045644835 (527h18m33.91672788s) [noise=true , address=45.91.116.9:4263 , piece_id=U2DLXXCX2NEHH7FMRROW3WB74RO7BNOVCJE55OHHBDSJESFQQWAQ, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.046817898 (527h18m33.917900944s) [noise=true , address=145.79.227.49:4338 , piece_id=JHGXIUN2XR6N7IUAXJRYKTKY3SHA4NKPM22RQK5UI2WUXTGT4RQQ, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.048385483 (527h18m33.919468528s) [noise=true , address=45.91.116.5:4297 , piece_id=SDMA6DBBTHSY2XRTGAVVLKBGVPITNYKNDVXWWB5WJTRQQHLO5SCQ, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.048945000 (527h18m33.920028045s) [noise=true , address=145.79.227.37:4567 , piece_id=QOACCYWCR5KLJMVZPTIXYMN7LZD6FHD5UDPF2CT3WLRKH7EYYKBQ, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.050550727 (527h18m33.921633772s) [noise=true , address=77.90.31.11:4554 , piece_id=D5VC6SJIH4ISSJ35UWA5NTE7QZTV6R4ZP5PY3LDRMRFBNDGV4LCQ, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.051751203 (527h18m33.922834248s) [noise=true , address=145.79.227.39:4014 , piece_id=FVJSRVM565PUJN4HZLBOFPI3M5EWDOFYY2UUUSTZSWZKN6USCULA, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.052569822 (527h18m33.923652857s) [noise=true , address=45.91.116.12:4226 , piece_id=2QIM5YKXSEYLEJEFAWDJGL75YGHAE5XPCS4RBDPLMPTRP3L4E53A, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:34.054092180 (527h18m33.925175225s) [noise=true , address=45.91.116.6:4973 , piece_id=MGASCARKZ6424UA3HAJUOI6QC2Z4H5PTG52YLUFM47NR4JSDMBKQ, seg_pos={0 0}, upload=462504]: Uploading piece... Dec 27 14:47:35.741527659 (527h18m35.612610694s) [upload=462504]: Next returned nil segment Dec 27 14:47:35.741541395 (527h18m35.61262444s) [upload=462504]: Waiting for error group managing segments... Dec 27 14:47:36.028778814 (527h18m35.899861859s) [noise=true , address=145.79.227.50:4374 , piece_id=PQBTCCNOFDPJ7UTUPGOXC42NOL5ZRPPLMVMTZR2O6YPZQLBT4PGA, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.055964650 (527h18m35.927047695s) [noise=true , address=145.79.227.37:4567 , piece_id=QOACCYWCR5KLJMVZPTIXYMN7LZD6FHD5UDPF2CT3WLRKH7EYYKBQ, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.056431579 (527h18m35.927514614s) [noise=true , address=145.79.227.39:4014 , piece_id=FVJSRVM565PUJN4HZLBOFPI3M5EWDOFYY2UUUSTZSWZKN6USCULA, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.076200563 (527h18m35.947283608s) [noise=true , address=145.79.227.41:4085 , piece_id=QBXCN372TIX2K3OOJ3R4IDEKGCPZJGJQWEGEQ3ISKZPE7OPSNWSA, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.078852502 (527h18m35.949935547s) [noise=true , address=145.79.227.49:4338 , piece_id=JHGXIUN2XR6N7IUAXJRYKTKY3SHA4NKPM22RQK5UI2WUXTGT4RQQ, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.278837388 (527h18m36.149920433s) [noise=true , address=45.91.116.5:4297 , piece_id=SDMA6DBBTHSY2XRTGAVVLKBGVPITNYKNDVXWWB5WJTRQQHLO5SCQ, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.287310383 (527h18m36.158393428s) [noise=true , address=45.91.116.9:4263 , piece_id=U2DLXXCX2NEHH7FMRROW3WB74RO7BNOVCJE55OHHBDSJESFQQWAQ, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.333665580 (527h18m36.204748616s) [noise=true , address=45.91.116.6:4973 , piece_id=MGASCARKZ6424UA3HAJUOI6QC2Z4H5PTG52YLUFM47NR4JSDMBKQ, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.348001368 (527h18m36.219084414s) [noise=true , address=77.90.31.11:4554 , piece_id=D5VC6SJIH4ISSJ35UWA5NTE7QZTV6R4ZP5PY3LDRMRFBNDGV4LCQ, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.362907472 (527h18m36.233990508s) [noise=true , address=77.90.31.18:4805 , piece_id=Y7FVDVV3IEDY4XDFI67XZAL7D2SFVKQSD47CYMJV7GWY4XRGUNFQ, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.484591630 (527h18m36.355674675s) [noise=true , address=77.90.31.15:4700 , piece_id=INWYR4O6ILASY5TLXUGH7AAJ7Z5BRFPTVWKWQTGGRD2OQPFQIFFA, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.526615280 (527h18m36.397698325s) [noise=true , address=45.91.116.10:4081 , piece_id=MFZ3F2O3KRQ7SAQWGNHVNLWPQUBFOVC75CZNZMTKDSHDC3ZBDCWA, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.655545496 (527h18m36.526628531s) [noise=true , address=77.90.31.17:4773 , piece_id=OUX67FKMVDHYXXUOGBRS26Z3RRQOQGZ75SER7LPUFW27DXU6GSUA, seg_pos={0 0}, upload=462504]: Done uploading piece. err: <nil> Dec 27 14:47:36.655565775 (527h18m36.52664882s) [seg_pos={0 0}, upload=462504]: Segment reached optimal threshold of 13 pieces. Dec 27 14:47:36.655737682 (527h18m36.526820717s) [noise=true , address=77.90.31.10:4519 , piece_id=BF564TG2HBB2USZEFRRO2OFHVOD4QNKHQISSDUCWFW55LRCKXK7Q, seg_pos={0 0}, upload=462504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 27 14:47:36.655745828 (527h18m36.526828863s) [noise=true , address=77.90.31.10:4519 , piece_id=BF564TG2HBB2USZEFRRO2OFHVOD4QNKHQISSDUCWFW55LRCKXK7Q, seg_pos={0 0}, upload=462504]: Long tail cancellation due to optimal threshold Dec 27 14:47:36.655826531 (527h18m36.526909596s) [noise=true , address=45.91.116.7:4009 , piece_id=SFSUC6FHT5CNQR4J3RYK36APS4G4AMBVXYEBKDX5XNRWXBASO5SQ, seg_pos={0 0}, upload=462504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 27 14:47:36.655836960 (527h18m36.526919996s) [noise=true , address=45.91.116.7:4009 , piece_id=SFSUC6FHT5CNQR4J3RYK36APS4G4AMBVXYEBKDX5XNRWXBASO5SQ, seg_pos={0 0}, upload=462504]: Long tail cancellation due to optimal threshold Dec 27 14:47:36.656007666 (527h18m36.527090711s) [noise=true , address=45.91.116.12:4226 , piece_id=2QIM5YKXSEYLEJEFAWDJGL75YGHAE5XPCS4RBDPLMPTRP3L4E53A, seg_pos={0 0}, upload=462504]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: read tcp 188.166.225.170:58942->45.91.116.12:4226: use of closed network connection; context canceled Dec 27 14:47:36.656013687 (527h18m36.527096722s) [noise=true , address=45.91.116.12:4226 , piece_id=2QIM5YKXSEYLEJEFAWDJGL75YGHAE5XPCS4RBDPLMPTRP3L4E53A, seg_pos={0 0}, upload=462504]: Long tail cancellation due to optimal threshold Dec 27 14:47:36.656028556 (527h18m36.527111591s) [seg_pos={0 0}, upload=462504]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 14:47:36.656126823 (527h18m36.527209858s) [upload=462504]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:47:36.656131532 (527h18m36.527214567s) [upload=462504]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:47:36.954703433 (527h18m36.825786478s) [upload=462504]: Done uploading segments... Dec 27 14:47:36.954724975 (527h18m36.82580801s) [upload=462504]: Upload finished. err: <nil> Duration: 30.003920421s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................X.............X..X...........X........] duration: 57.129074575s trace: https://jaeger.dev.storj.io/trace/4d753dda46642d0d FAIL (2025-12-27 14:48:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd, Piece ID: 5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA) context deadline exceeded; piecestore: (Node ID: 12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ, Piece ID: ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ) context deadline exceeded; piecestore: (Node ID: 12cth5bdi5iAWhHcLxqN9c7GLdheS7SLQ3x5fEHG1MdgdXWFdD6, Piece ID: OCZIJHDS4AESHNCHUJIBVN47X336BIKXBLIQ4BVYSF7LAPWIS3KA) context deadline exceeded ======LOGS====== 660 (527h10m58.618722705s) [noise=true , address=77.90.31.19:4860 , piece_id=5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748001970 (527h10m58.619085005s) [noise=true , address=145.79.227.44:4205 , piece_id=HVNDI6C4SQZYPZIEEP7TC5RW6YNW5WGFDOJKQIIAN3UH3VGDXNIA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748340274 (527h10m58.619423319s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748682386 (527h10m58.619765421s) [noise=true , address=77.90.31.10:4530 , piece_id=K22R6V3CWZWVFKROB23NKYFC5R2Z5LT4KSXMTXFQUUROJKCJWESA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748994651 (527h10m58.620077696s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.749319318 (527h10m58.620402363s) [noise=true , address=145.79.227.40:4060 , piece_id=OADO7RXPLZ4NLGUC2CCXNJQ6OEMU5JWTXCRY7WFWYKKGLMVU6YBA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.749662011 (527h10m58.620745046s) [noise=true , address=77.90.31.12:4601 , piece_id=LCBL5PH44YSNTX6W3UBMGJAGSOTBB6CKMBBD7E3GKDFQBUXMR3AA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.750250261 (527h10m58.621333296s) [noise=true , address=145.79.227.46:4633 , piece_id=U6GO7WKUMICS7QYSJZXZ6ZB33S2MYCV5JGEPVHEG7BHX5HGZL3XA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.750568958 (527h10m58.621651993s) [noise=true , address=145.79.227.23:4579 , piece_id=A3LOK5DMHEU3G3C6XYC2CWLOL7U4XJFXK4RV4A3PH4AXRHTA4EGQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.751689872 (527h10m58.622772917s) [noise=true , address=45.91.116.8:4049 , piece_id=ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.751758763 (527h10m58.622841809s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.753063619 (527h10m58.624146654s) [noise=true , address=145.79.227.36:4537 , piece_id=TMCDI7JYHU6MAPAGTLJTB2GSXMSBPJZJSPZWGAFF5I4VGEGNOMUQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.753568099 (527h10m58.624651134s) [noise=true , address=145.79.227.8:4337 , piece_id=BC5P5Y75VGZDTNDWEOC27UTK3TMWFRT7BNKQTJAYYHPYLYRXV2VQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.754895386 (527h10m58.625978421s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.755999769 (527h10m58.627082804s) [noise=true , address=77.90.31.21:4893 , piece_id=WYF4STSQ27E7KEMRIXKC4ILRHKN4UCNGH5NJFZ5SHOBKW3FNUCHA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.757486991 (527h10m58.628570037s) [noise=true , address=64.49.13.154:4933 , piece_id=V7STA7QTESDS5FDALZIPURQRCO6X4MONA2PTU5RIJHM34P5BQTNA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:40:00.120776880 (527h10m59.991859915s) [upload=462397]: Next returned nil segment Dec 27 14:40:00.120795044 (527h10m59.991878079s) [upload=462397]: Waiting for error group managing segments... Dec 27 14:40:00.475194542 (527h11m0.346277587s) [noise=true , address=64.49.13.154:4933 , piece_id=V7STA7QTESDS5FDALZIPURQRCO6X4MONA2PTU5RIJHM34P5BQTNA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.626897129 (527h11m0.497980174s) [noise=true , address=145.79.227.46:4633 , piece_id=U6GO7WKUMICS7QYSJZXZ6ZB33S2MYCV5JGEPVHEG7BHX5HGZL3XA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.634206858 (527h11m0.505289903s) [noise=true , address=145.79.227.34:4927 , piece_id=3PJB2HW34B5N5LADQYZ3FPNEIWBGKKGJEGADMPJIMWRG6HGQG36Q, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.636936777 (527h11m0.508019822s) [noise=true , address=145.79.227.36:4537 , piece_id=TMCDI7JYHU6MAPAGTLJTB2GSXMSBPJZJSPZWGAFF5I4VGEGNOMUQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.650341221 (527h11m0.521424266s) [noise=true , address=145.79.227.8:4337 , piece_id=BC5P5Y75VGZDTNDWEOC27UTK3TMWFRT7BNKQTJAYYHPYLYRXV2VQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.662509752 (527h11m0.533592797s) [noise=true , address=145.79.227.23:4579 , piece_id=A3LOK5DMHEU3G3C6XYC2CWLOL7U4XJFXK4RV4A3PH4AXRHTA4EGQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.820994414 (527h11m0.692077449s) [noise=true , address=77.90.31.12:4601 , piece_id=LCBL5PH44YSNTX6W3UBMGJAGSOTBB6CKMBBD7E3GKDFQBUXMR3AA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.908728394 (527h11m0.779811429s) [noise=true , address=45.91.116.8:4049 , piece_id=ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.921937597 (527h11m0.793020642s) [noise=true , address=77.90.31.10:4530 , piece_id=K22R6V3CWZWVFKROB23NKYFC5R2Z5LT4KSXMTXFQUUROJKCJWESA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.180318328 (527h11m1.051401374s) [noise=true , address=77.90.31.14:4683 , piece_id=OCZIJHDS4AESHNCHUJIBVN47X336BIKXBLIQ4BVYSF7LAPWIS3KA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.467385073 (527h11m1.338468118s) [noise=true , address=77.90.31.19:4860 , piece_id=5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.506047844 (527h11m1.377130919s) [noise=true , address=145.79.227.44:4205 , piece_id=HVNDI6C4SQZYPZIEEP7TC5RW6YNW5WGFDOJKQIIAN3UH3VGDXNIA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.551036449 (527h11m1.422119494s) [noise=true , address=145.79.227.40:4060 , piece_id=OADO7RXPLZ4NLGUC2CCXNJQ6OEMU5JWTXCRY7WFWYKKGLMVU6YBA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.567791785 (527h11m1.43887482s) [noise=true , address=77.90.31.21:4893 , piece_id=WYF4STSQ27E7KEMRIXKC4ILRHKN4UCNGH5NJFZ5SHOBKW3FNUCHA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.567835187 (527h11m1.438918232s) [seg_pos={0 0}, upload=462397]: Segment reached optimal threshold of 14 pieces. Dec 27 14:40:01.568097526 (527h11m1.439180581s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568109259 (527h11m1.439192304s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568506055 (527h11m1.4395891s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568516865 (527h11m1.43959991s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568673093 (527h11m1.439756148s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection Dec 27 14:40:01.568680948 (527h11m1.439763993s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568844680 (527h11m1.439927725s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FtB4bKTHntdUQxF8G5o9nT73oS2xB7fgD44BkRfKDNFyZnKCk): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568852474 (527h11m1.439935519s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568872633 (527h11m1.439955678s) [seg_pos={0 0}, upload=462397]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 14:40:01.568960319 (527h11m1.440043365s) [upload=462397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:40:01.568966050 (527h11m1.440049095s) [upload=462397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:40:01.862585703 (527h11m1.733668748s) [upload=462397]: Done uploading segments... Dec 27 14:40:01.862613346 (527h11m1.733696401s) [upload=462397]: Upload finished. err: <nil> Duration: 57.129099342s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [XXXXXXX.XXXXXX.XX.XXXXXX.XXXX.XXXXXXXXXXXX..XXXXXXX.....XXX.] duration: 5.001590219s trace: https://jaeger.dev.storj.io/trace/165897763121c0e FAIL (2025-12-27 14:47:47 UTC) orbiter warning: context deadline exceeded Duration: 5.001613372s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 7.616387523s trace: https://jaeger.dev.storj.io/trace/2f00e104e333c43f FAIL (2025-12-27 14:47:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== }, upload=462172]: Uploading piece... Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments... Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces. Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments... Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil> Duration: 7.616415296s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.XXX..X...XX.............X..X....X.........X.X....X...X.....] duration: 6.125519984s trace: https://jaeger.dev.storj.io/trace/4f06cb127c622fe FAIL (2025-12-27 14:47:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6163s) [upload=462438]: Uploading segments... Dec 27 14:42:49.799089041 (527h13m49.670172096s) [upload=462438]: Got next segment. Inline: false Dec 27 14:42:49.799144557 (527h13m49.670227602s) [upload=462438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 14:42:49.799150438 (527h13m49.670233483s) [upload=462438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 14:42:50.043758773 (527h13m49.914841808s) [seg_pos={0 0}, upload=462438]: Begin upload segment... Dec 27 14:42:50.043802957 (527h13m49.914885992s) [seg_pos={0 0}, upload=462438]: Done begin upload segment. Dec 27 14:42:50.043842984 (527h13m49.914926029s) [noise=true , address=77.90.31.14:4664 , piece_id=4IP7OZYCQ2LPVW4PVP66PTSKYXYSIXHP25BMWWUSUUEI6SXXMOHQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.044003700 (527h13m49.915086745s) [noise=true , address=145.79.227.50:4375 , piece_id=PO4M5RTMTGXBCVB67CC3NPMYCNI354WGADEJJAIDMBQZZT62JCVQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.044910416 (527h13m49.915993491s) [noise=true , address=45.91.116.6:4975 , piece_id=3HG4RWJFNSJJ3LZDRE5OLAFQILGCPFQXAH4YQCPRBPSGXP7C26GQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.045314165 (527h13m49.9163972s) [noise=true , address=45.91.116.5:4299 , piece_id=YAR5WZGPTKC3UAGTV25YD2LZX3G2CQGWPTYCB3OBJCK2SZEN7MSA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.046014649 (527h13m49.917097684s) [noise=true , address=45.91.116.10:4083 , piece_id=WNEYFFRZ3EQIEP7LQZMAYY3B3KVK7DTYUG6LB4QJ2J5CWGKADAXA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.046429319 (527h13m49.917512364s) [noise=true , address=145.79.227.16:4912 , piece_id=OVJ6A3TWB35UB2KAKLW5MB3DX3VHOWWPF7AMFVD3JVRQPZORGU3Q, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.047612652 (527h13m49.918695697s) [noise=true , address=145.79.227.3:4149 , piece_id=X4KQZTA7BS3HX2HUJCWNUKV4ZC673VTJRZUX6GESZ7M6EKXYTTMQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.049355891 (527h13m49.920438967s) [noise=true , address=145.79.227.29:4941 , piece_id=PR2XNGAX4EMGVZVXZHUMXAMHK7L6TWP6AIO2SABB5JZ5YHVK2LAA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.050241948 (527h13m49.921324993s) [noise=true , address=145.79.227.2:4114 , piece_id=XFVTYCSRMRC5RAQF3X4V6VDHLUDMP7ILK266R36L62DM47PUEHHA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.053153603 (527h13m49.924236648s) [noise=true , address=45.91.116.11:4118 , piece_id=Q3XCMOCYATPS6J6QTUXBQ3ZMVWNR6BDI5NMB7ALKBPZP2V6WKTKQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.053572400 (527h13m49.924655435s) [noise=true , address=77.90.31.10:4517 , piece_id=CJYRTQJM6XQD563UV3WJCI5ZGFUIO2ZAOQ42E4OOAPUBT7GJA3TQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.055566928 (527h13m49.926649963s) [noise=true , address=145.79.227.46:4603 , piece_id=RPYXKRL6XOHOCVZI6HLSFK2ZY3MFGTC3YMPVE3CJZ7QNUIH4KTLQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.056885168 (527h13m49.927968203s) [noise=true , address=45.91.116.8:4048 , piece_id=NPXO5LTRCVZ7KJL42FVT2ZWOSSS7PXAQVJHF7OYPWK4PGPNNGW7A, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.058750811 (527h13m49.929833856s) [noise=true , address=45.91.116.7:4009 , piece_id=34CGGMU5V72HAVH66PKI2A6IYYUGHYDN6CXYBI3GSDJTDHIJBPCA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.060156948 (527h13m49.931239993s) [noise=true , address=145.79.227.4:4186 , piece_id=YRPDW2GS3YDTGUB4VJRWF2S3W7434K4JQWX3UOEAVFR676Z2CDNA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.061310124 (527h13m49.932393169s) [noise=true , address=45.91.116.14:4189 , piece_id=VCTSVCCJ5MHIIDWKHZPWENEEODBT6JMB4FLSWYXSWZIEH6UZV2NA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:51.763514444 (527h13m51.634597489s) [upload=462438]: Next returned nil segment Dec 27 14:42:51.763530965 (527h13m51.634614s) [upload=462438]: Waiting for error group managing segments... Dec 27 14:42:52.048902352 (527h13m51.919985397s) [noise=true , address=145.79.227.16:4912 , piece_id=OVJ6A3TWB35UB2KAKLW5MB3DX3VHOWWPF7AMFVD3JVRQPZORGU3Q, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.063639583 (527h13m51.934722639s) [noise=true , address=145.79.227.4:4186 , piece_id=YRPDW2GS3YDTGUB4VJRWF2S3W7434K4JQWX3UOEAVFR676Z2CDNA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.069432584 (527h13m51.940515619s) [noise=true , address=145.79.227.50:4375 , piece_id=PO4M5RTMTGXBCVB67CC3NPMYCNI354WGADEJJAIDMBQZZT62JCVQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.069628026 (527h13m51.940711072s) [noise=true , address=145.79.227.2:4114 , piece_id=XFVTYCSRMRC5RAQF3X4V6VDHLUDMP7ILK266R36L62DM47PUEHHA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.083496475 (527h13m51.95457953s) [noise=true , address=145.79.227.46:4603 , piece_id=RPYXKRL6XOHOCVZI6HLSFK2ZY3MFGTC3YMPVE3CJZ7QNUIH4KTLQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.086381027 (527h13m51.957464062s) [noise=true , address=145.79.227.3:4149 , piece_id=X4KQZTA7BS3HX2HUJCWNUKV4ZC673VTJRZUX6GESZ7M6EKXYTTMQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.091852015 (527h13m51.96293506s) [noise=true , address=145.79.227.29:4941 , piece_id=PR2XNGAX4EMGVZVXZHUMXAMHK7L6TWP6AIO2SABB5JZ5YHVK2LAA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.254422116 (527h13m52.125505151s) [noise=true , address=77.90.31.10:4517 , piece_id=CJYRTQJM6XQD563UV3WJCI5ZGFUIO2ZAOQ42E4OOAPUBT7GJA3TQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.316266098 (527h13m52.187349133s) [noise=true , address=45.91.116.8:4048 , piece_id=NPXO5LTRCVZ7KJL42FVT2ZWOSSS7PXAQVJHF7OYPWK4PGPNNGW7A, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.324335654 (527h13m52.195418699s) [noise=true , address=45.91.116.10:4083 , piece_id=WNEYFFRZ3EQIEP7LQZMAYY3B3KVK7DTYUG6LB4QJ2J5CWGKADAXA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.456401440 (527h13m52.327484475s) [noise=true , address=45.91.116.14:4189 , piece_id=VCTSVCCJ5MHIIDWKHZPWENEEODBT6JMB4FLSWYXSWZIEH6UZV2NA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.615789081 (527h13m52.486872126s) [noise=true , address=45.91.116.7:4009 , piece_id=34CGGMU5V72HAVH66PKI2A6IYYUGHYDN6CXYBI3GSDJTDHIJBPCA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.936703357 (527h13m52.807786402s) [noise=true , address=77.90.31.14:4664 , piece_id=4IP7OZYCQ2LPVW4PVP66PTSKYXYSIXHP25BMWWUSUUEI6SXXMOHQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.936743273 (527h13m52.807826318s) [seg_pos={0 0}, upload=462438]: Segment reached optimal threshold of 13 pieces. Dec 27 14:42:52.942008298 (527h13m52.813091344s) [noise=true , address=45.91.116.11:4118 , piece_id=Q3XCMOCYATPS6J6QTUXBQ3ZMVWNR6BDI5NMB7ALKBPZP2V6WKTKQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:44872->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:44872->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44872->45.91.116.11:4118: use of closed network connection Dec 27 14:42:52.942024039 (527h13m52.813107074s) [noise=true , address=45.91.116.11:4118 , piece_id=Q3XCMOCYATPS6J6QTUXBQ3ZMVWNR6BDI5NMB7ALKBPZP2V6WKTKQ, seg_pos={0 0}, upload=462438]: Long tail cancellation due to optimal threshold Dec 27 14:42:52.945792765 (527h13m52.81687585s) [noise=true , address=45.91.116.6:4975 , piece_id=3HG4RWJFNSJJ3LZDRE5OLAFQILGCPFQXAH4YQCPRBPSGXP7C26GQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:41890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 14:42:52.945882115 (527h13m52.81696516s) [noise=true , address=45.91.116.6:4975 , piece_id=3HG4RWJFNSJJ3LZDRE5OLAFQILGCPFQXAH4YQCPRBPSGXP7C26GQ, seg_pos={0 0}, upload=462438]: Long tail cancellation due to optimal threshold Dec 27 14:42:52.946209569 (527h13m52.817292624s) [noise=true , address=45.91.116.5:4299 , piece_id=YAR5WZGPTKC3UAGTV25YD2LZX3G2CQGWPTYCB3OBJCK2SZEN7MSA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: read tcp 188.166.225.170:43062->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 14:42:52.946218666 (527h13m52.817301701s) [noise=true , address=45.91.116.5:4299 , piece_id=YAR5WZGPTKC3UAGTV25YD2LZX3G2CQGWPTYCB3OBJCK2SZEN7MSA, seg_pos={0 0}, upload=462438]: Long tail cancellation due to optimal threshold Dec 27 14:42:52.946236721 (527h13m52.817319766s) [seg_pos={0 0}, upload=462438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 14:42:52.946338554 (527h13m52.817421599s) [upload=462438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:42:52.946346790 (527h13m52.817429835s) [upload=462438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:42:53.234888003 (527h13m53.105971048s) [upload=462438]: Done uploading segments... Dec 27 14:42:53.234911567 (527h13m53.105994612s) [upload=462438]: Upload finished. err: <nil> Duration: 6.12555011s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 16.784638922s trace: https://jaeger.dev.storj.io/trace/2f00e104e333c43f FAIL (2025-12-27 14:47:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== }, upload=462172]: Uploading piece... Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments... Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces. Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments... Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil> Duration: 16.784687063s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [.XXXXXXX.XXXXXX.XX.XXXXXX.XXXX.XXXXXXXXXXXX..XXXXXXX.....XXX] duration: 5.001293515s trace: https://jaeger.dev.storj.io/trace/165897763121c0e FAIL (2025-12-27 14:46:44 UTC) orbiter warning: context deadline exceeded Duration: 5.001314633s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................X.............X..X...........X.......] duration: 45.691985669s trace: https://jaeger.dev.storj.io/trace/4d753dda46642d0d FAIL (2025-12-27 14:46:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd, Piece ID: 5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA) context deadline exceeded; piecestore: (Node ID: 12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ, Piece ID: ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ) context deadline exceeded; piecestore: (Node ID: 12cth5bdi5iAWhHcLxqN9c7GLdheS7SLQ3x5fEHG1MdgdXWFdD6, Piece ID: OCZIJHDS4AESHNCHUJIBVN47X336BIKXBLIQ4BVYSF7LAPWIS3KA) context deadline exceeded ======LOGS====== 660 (527h10m58.618722705s) [noise=true , address=77.90.31.19:4860 , piece_id=5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748001970 (527h10m58.619085005s) [noise=true , address=145.79.227.44:4205 , piece_id=HVNDI6C4SQZYPZIEEP7TC5RW6YNW5WGFDOJKQIIAN3UH3VGDXNIA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748340274 (527h10m58.619423319s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748682386 (527h10m58.619765421s) [noise=true , address=77.90.31.10:4530 , piece_id=K22R6V3CWZWVFKROB23NKYFC5R2Z5LT4KSXMTXFQUUROJKCJWESA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748994651 (527h10m58.620077696s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.749319318 (527h10m58.620402363s) [noise=true , address=145.79.227.40:4060 , piece_id=OADO7RXPLZ4NLGUC2CCXNJQ6OEMU5JWTXCRY7WFWYKKGLMVU6YBA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.749662011 (527h10m58.620745046s) [noise=true , address=77.90.31.12:4601 , piece_id=LCBL5PH44YSNTX6W3UBMGJAGSOTBB6CKMBBD7E3GKDFQBUXMR3AA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.750250261 (527h10m58.621333296s) [noise=true , address=145.79.227.46:4633 , piece_id=U6GO7WKUMICS7QYSJZXZ6ZB33S2MYCV5JGEPVHEG7BHX5HGZL3XA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.750568958 (527h10m58.621651993s) [noise=true , address=145.79.227.23:4579 , piece_id=A3LOK5DMHEU3G3C6XYC2CWLOL7U4XJFXK4RV4A3PH4AXRHTA4EGQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.751689872 (527h10m58.622772917s) [noise=true , address=45.91.116.8:4049 , piece_id=ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.751758763 (527h10m58.622841809s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.753063619 (527h10m58.624146654s) [noise=true , address=145.79.227.36:4537 , piece_id=TMCDI7JYHU6MAPAGTLJTB2GSXMSBPJZJSPZWGAFF5I4VGEGNOMUQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.753568099 (527h10m58.624651134s) [noise=true , address=145.79.227.8:4337 , piece_id=BC5P5Y75VGZDTNDWEOC27UTK3TMWFRT7BNKQTJAYYHPYLYRXV2VQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.754895386 (527h10m58.625978421s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.755999769 (527h10m58.627082804s) [noise=true , address=77.90.31.21:4893 , piece_id=WYF4STSQ27E7KEMRIXKC4ILRHKN4UCNGH5NJFZ5SHOBKW3FNUCHA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.757486991 (527h10m58.628570037s) [noise=true , address=64.49.13.154:4933 , piece_id=V7STA7QTESDS5FDALZIPURQRCO6X4MONA2PTU5RIJHM34P5BQTNA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:40:00.120776880 (527h10m59.991859915s) [upload=462397]: Next returned nil segment Dec 27 14:40:00.120795044 (527h10m59.991878079s) [upload=462397]: Waiting for error group managing segments... Dec 27 14:40:00.475194542 (527h11m0.346277587s) [noise=true , address=64.49.13.154:4933 , piece_id=V7STA7QTESDS5FDALZIPURQRCO6X4MONA2PTU5RIJHM34P5BQTNA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.626897129 (527h11m0.497980174s) [noise=true , address=145.79.227.46:4633 , piece_id=U6GO7WKUMICS7QYSJZXZ6ZB33S2MYCV5JGEPVHEG7BHX5HGZL3XA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.634206858 (527h11m0.505289903s) [noise=true , address=145.79.227.34:4927 , piece_id=3PJB2HW34B5N5LADQYZ3FPNEIWBGKKGJEGADMPJIMWRG6HGQG36Q, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.636936777 (527h11m0.508019822s) [noise=true , address=145.79.227.36:4537 , piece_id=TMCDI7JYHU6MAPAGTLJTB2GSXMSBPJZJSPZWGAFF5I4VGEGNOMUQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.650341221 (527h11m0.521424266s) [noise=true , address=145.79.227.8:4337 , piece_id=BC5P5Y75VGZDTNDWEOC27UTK3TMWFRT7BNKQTJAYYHPYLYRXV2VQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.662509752 (527h11m0.533592797s) [noise=true , address=145.79.227.23:4579 , piece_id=A3LOK5DMHEU3G3C6XYC2CWLOL7U4XJFXK4RV4A3PH4AXRHTA4EGQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.820994414 (527h11m0.692077449s) [noise=true , address=77.90.31.12:4601 , piece_id=LCBL5PH44YSNTX6W3UBMGJAGSOTBB6CKMBBD7E3GKDFQBUXMR3AA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.908728394 (527h11m0.779811429s) [noise=true , address=45.91.116.8:4049 , piece_id=ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.921937597 (527h11m0.793020642s) [noise=true , address=77.90.31.10:4530 , piece_id=K22R6V3CWZWVFKROB23NKYFC5R2Z5LT4KSXMTXFQUUROJKCJWESA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.180318328 (527h11m1.051401374s) [noise=true , address=77.90.31.14:4683 , piece_id=OCZIJHDS4AESHNCHUJIBVN47X336BIKXBLIQ4BVYSF7LAPWIS3KA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.467385073 (527h11m1.338468118s) [noise=true , address=77.90.31.19:4860 , piece_id=5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.506047844 (527h11m1.377130919s) [noise=true , address=145.79.227.44:4205 , piece_id=HVNDI6C4SQZYPZIEEP7TC5RW6YNW5WGFDOJKQIIAN3UH3VGDXNIA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.551036449 (527h11m1.422119494s) [noise=true , address=145.79.227.40:4060 , piece_id=OADO7RXPLZ4NLGUC2CCXNJQ6OEMU5JWTXCRY7WFWYKKGLMVU6YBA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.567791785 (527h11m1.43887482s) [noise=true , address=77.90.31.21:4893 , piece_id=WYF4STSQ27E7KEMRIXKC4ILRHKN4UCNGH5NJFZ5SHOBKW3FNUCHA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.567835187 (527h11m1.438918232s) [seg_pos={0 0}, upload=462397]: Segment reached optimal threshold of 14 pieces. Dec 27 14:40:01.568097526 (527h11m1.439180581s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568109259 (527h11m1.439192304s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568506055 (527h11m1.4395891s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568516865 (527h11m1.43959991s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568673093 (527h11m1.439756148s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection Dec 27 14:40:01.568680948 (527h11m1.439763993s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568844680 (527h11m1.439927725s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FtB4bKTHntdUQxF8G5o9nT73oS2xB7fgD44BkRfKDNFyZnKCk): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568852474 (527h11m1.439935519s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568872633 (527h11m1.439955678s) [seg_pos={0 0}, upload=462397]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 14:40:01.568960319 (527h11m1.440043365s) [upload=462397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:40:01.568966050 (527h11m1.440049095s) [upload=462397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:40:01.862585703 (527h11m1.733668748s) [upload=462397]: Done uploading segments... Dec 27 14:40:01.862613346 (527h11m1.733696401s) [upload=462397]: Upload finished. err: <nil> Duration: 45.692000206s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.XXX..X...XX.............X..X....X.........X.X....X...X....] duration: 6.696570955s trace: https://jaeger.dev.storj.io/trace/4f06cb127c622fe FAIL (2025-12-27 14:46:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6163s) [upload=462438]: Uploading segments... Dec 27 14:42:49.799089041 (527h13m49.670172096s) [upload=462438]: Got next segment. Inline: false Dec 27 14:42:49.799144557 (527h13m49.670227602s) [upload=462438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 14:42:49.799150438 (527h13m49.670233483s) [upload=462438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 14:42:50.043758773 (527h13m49.914841808s) [seg_pos={0 0}, upload=462438]: Begin upload segment... Dec 27 14:42:50.043802957 (527h13m49.914885992s) [seg_pos={0 0}, upload=462438]: Done begin upload segment. Dec 27 14:42:50.043842984 (527h13m49.914926029s) [noise=true , address=77.90.31.14:4664 , piece_id=4IP7OZYCQ2LPVW4PVP66PTSKYXYSIXHP25BMWWUSUUEI6SXXMOHQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.044003700 (527h13m49.915086745s) [noise=true , address=145.79.227.50:4375 , piece_id=PO4M5RTMTGXBCVB67CC3NPMYCNI354WGADEJJAIDMBQZZT62JCVQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.044910416 (527h13m49.915993491s) [noise=true , address=45.91.116.6:4975 , piece_id=3HG4RWJFNSJJ3LZDRE5OLAFQILGCPFQXAH4YQCPRBPSGXP7C26GQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.045314165 (527h13m49.9163972s) [noise=true , address=45.91.116.5:4299 , piece_id=YAR5WZGPTKC3UAGTV25YD2LZX3G2CQGWPTYCB3OBJCK2SZEN7MSA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.046014649 (527h13m49.917097684s) [noise=true , address=45.91.116.10:4083 , piece_id=WNEYFFRZ3EQIEP7LQZMAYY3B3KVK7DTYUG6LB4QJ2J5CWGKADAXA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.046429319 (527h13m49.917512364s) [noise=true , address=145.79.227.16:4912 , piece_id=OVJ6A3TWB35UB2KAKLW5MB3DX3VHOWWPF7AMFVD3JVRQPZORGU3Q, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.047612652 (527h13m49.918695697s) [noise=true , address=145.79.227.3:4149 , piece_id=X4KQZTA7BS3HX2HUJCWNUKV4ZC673VTJRZUX6GESZ7M6EKXYTTMQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.049355891 (527h13m49.920438967s) [noise=true , address=145.79.227.29:4941 , piece_id=PR2XNGAX4EMGVZVXZHUMXAMHK7L6TWP6AIO2SABB5JZ5YHVK2LAA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.050241948 (527h13m49.921324993s) [noise=true , address=145.79.227.2:4114 , piece_id=XFVTYCSRMRC5RAQF3X4V6VDHLUDMP7ILK266R36L62DM47PUEHHA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.053153603 (527h13m49.924236648s) [noise=true , address=45.91.116.11:4118 , piece_id=Q3XCMOCYATPS6J6QTUXBQ3ZMVWNR6BDI5NMB7ALKBPZP2V6WKTKQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.053572400 (527h13m49.924655435s) [noise=true , address=77.90.31.10:4517 , piece_id=CJYRTQJM6XQD563UV3WJCI5ZGFUIO2ZAOQ42E4OOAPUBT7GJA3TQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.055566928 (527h13m49.926649963s) [noise=true , address=145.79.227.46:4603 , piece_id=RPYXKRL6XOHOCVZI6HLSFK2ZY3MFGTC3YMPVE3CJZ7QNUIH4KTLQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.056885168 (527h13m49.927968203s) [noise=true , address=45.91.116.8:4048 , piece_id=NPXO5LTRCVZ7KJL42FVT2ZWOSSS7PXAQVJHF7OYPWK4PGPNNGW7A, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.058750811 (527h13m49.929833856s) [noise=true , address=45.91.116.7:4009 , piece_id=34CGGMU5V72HAVH66PKI2A6IYYUGHYDN6CXYBI3GSDJTDHIJBPCA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.060156948 (527h13m49.931239993s) [noise=true , address=145.79.227.4:4186 , piece_id=YRPDW2GS3YDTGUB4VJRWF2S3W7434K4JQWX3UOEAVFR676Z2CDNA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.061310124 (527h13m49.932393169s) [noise=true , address=45.91.116.14:4189 , piece_id=VCTSVCCJ5MHIIDWKHZPWENEEODBT6JMB4FLSWYXSWZIEH6UZV2NA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:51.763514444 (527h13m51.634597489s) [upload=462438]: Next returned nil segment Dec 27 14:42:51.763530965 (527h13m51.634614s) [upload=462438]: Waiting for error group managing segments... Dec 27 14:42:52.048902352 (527h13m51.919985397s) [noise=true , address=145.79.227.16:4912 , piece_id=OVJ6A3TWB35UB2KAKLW5MB3DX3VHOWWPF7AMFVD3JVRQPZORGU3Q, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.063639583 (527h13m51.934722639s) [noise=true , address=145.79.227.4:4186 , piece_id=YRPDW2GS3YDTGUB4VJRWF2S3W7434K4JQWX3UOEAVFR676Z2CDNA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.069432584 (527h13m51.940515619s) [noise=true , address=145.79.227.50:4375 , piece_id=PO4M5RTMTGXBCVB67CC3NPMYCNI354WGADEJJAIDMBQZZT62JCVQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.069628026 (527h13m51.940711072s) [noise=true , address=145.79.227.2:4114 , piece_id=XFVTYCSRMRC5RAQF3X4V6VDHLUDMP7ILK266R36L62DM47PUEHHA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.083496475 (527h13m51.95457953s) [noise=true , address=145.79.227.46:4603 , piece_id=RPYXKRL6XOHOCVZI6HLSFK2ZY3MFGTC3YMPVE3CJZ7QNUIH4KTLQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.086381027 (527h13m51.957464062s) [noise=true , address=145.79.227.3:4149 , piece_id=X4KQZTA7BS3HX2HUJCWNUKV4ZC673VTJRZUX6GESZ7M6EKXYTTMQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.091852015 (527h13m51.96293506s) [noise=true , address=145.79.227.29:4941 , piece_id=PR2XNGAX4EMGVZVXZHUMXAMHK7L6TWP6AIO2SABB5JZ5YHVK2LAA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.254422116 (527h13m52.125505151s) [noise=true , address=77.90.31.10:4517 , piece_id=CJYRTQJM6XQD563UV3WJCI5ZGFUIO2ZAOQ42E4OOAPUBT7GJA3TQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.316266098 (527h13m52.187349133s) [noise=true , address=45.91.116.8:4048 , piece_id=NPXO5LTRCVZ7KJL42FVT2ZWOSSS7PXAQVJHF7OYPWK4PGPNNGW7A, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.324335654 (527h13m52.195418699s) [noise=true , address=45.91.116.10:4083 , piece_id=WNEYFFRZ3EQIEP7LQZMAYY3B3KVK7DTYUG6LB4QJ2J5CWGKADAXA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.456401440 (527h13m52.327484475s) [noise=true , address=45.91.116.14:4189 , piece_id=VCTSVCCJ5MHIIDWKHZPWENEEODBT6JMB4FLSWYXSWZIEH6UZV2NA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.615789081 (527h13m52.486872126s) [noise=true , address=45.91.116.7:4009 , piece_id=34CGGMU5V72HAVH66PKI2A6IYYUGHYDN6CXYBI3GSDJTDHIJBPCA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.936703357 (527h13m52.807786402s) [noise=true , address=77.90.31.14:4664 , piece_id=4IP7OZYCQ2LPVW4PVP66PTSKYXYSIXHP25BMWWUSUUEI6SXXMOHQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.936743273 (527h13m52.807826318s) [seg_pos={0 0}, upload=462438]: Segment reached optimal threshold of 13 pieces. Dec 27 14:42:52.942008298 (527h13m52.813091344s) [noise=true , address=45.91.116.11:4118 , piece_id=Q3XCMOCYATPS6J6QTUXBQ3ZMVWNR6BDI5NMB7ALKBPZP2V6WKTKQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:44872->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:44872->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44872->45.91.116.11:4118: use of closed network connection Dec 27 14:42:52.942024039 (527h13m52.813107074s) [noise=true , address=45.91.116.11:4118 , piece_id=Q3XCMOCYATPS6J6QTUXBQ3ZMVWNR6BDI5NMB7ALKBPZP2V6WKTKQ, seg_pos={0 0}, upload=462438]: Long tail cancellation due to optimal threshold Dec 27 14:42:52.945792765 (527h13m52.81687585s) [noise=true , address=45.91.116.6:4975 , piece_id=3HG4RWJFNSJJ3LZDRE5OLAFQILGCPFQXAH4YQCPRBPSGXP7C26GQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:41890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 14:42:52.945882115 (527h13m52.81696516s) [noise=true , address=45.91.116.6:4975 , piece_id=3HG4RWJFNSJJ3LZDRE5OLAFQILGCPFQXAH4YQCPRBPSGXP7C26GQ, seg_pos={0 0}, upload=462438]: Long tail cancellation due to optimal threshold Dec 27 14:42:52.946209569 (527h13m52.817292624s) [noise=true , address=45.91.116.5:4299 , piece_id=YAR5WZGPTKC3UAGTV25YD2LZX3G2CQGWPTYCB3OBJCK2SZEN7MSA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: read tcp 188.166.225.170:43062->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 14:42:52.946218666 (527h13m52.817301701s) [noise=true , address=45.91.116.5:4299 , piece_id=YAR5WZGPTKC3UAGTV25YD2LZX3G2CQGWPTYCB3OBJCK2SZEN7MSA, seg_pos={0 0}, upload=462438]: Long tail cancellation due to optimal threshold Dec 27 14:42:52.946236721 (527h13m52.817319766s) [seg_pos={0 0}, upload=462438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 14:42:52.946338554 (527h13m52.817421599s) [upload=462438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:42:52.946346790 (527h13m52.817429835s) [upload=462438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:42:53.234888003 (527h13m53.105971048s) [upload=462438]: Done uploading segments... Dec 27 14:42:53.234911567 (527h13m53.105994612s) [upload=462438]: Upload finished. err: <nil> Duration: 6.696602003s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 7.774036514s trace: https://jaeger.dev.storj.io/trace/2f00e104e333c43f FAIL (2025-12-27 14:46:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== }, upload=462172]: Uploading piece... Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments... Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces. Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments... Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil> Duration: 7.774059899s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [X.XXXXXXX.XXXXXX.XX.XXXXXX.XXXX.XXXXXXXXXXXX..XXXXXXX.....XX] duration: 5.001584918s trace: https://jaeger.dev.storj.io/trace/516918b775277bb FAIL (2025-12-27 14:46:03 UTC) orbiter warning: context deadline exceeded Duration: 5.001606921s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................X.............X..X...........X......] duration: 48.806231148s trace: https://jaeger.dev.storj.io/trace/4d753dda46642d0d FAIL (2025-12-27 14:45:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd, Piece ID: 5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA) context deadline exceeded; piecestore: (Node ID: 12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ, Piece ID: ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ) context deadline exceeded; piecestore: (Node ID: 12cth5bdi5iAWhHcLxqN9c7GLdheS7SLQ3x5fEHG1MdgdXWFdD6, Piece ID: OCZIJHDS4AESHNCHUJIBVN47X336BIKXBLIQ4BVYSF7LAPWIS3KA) context deadline exceeded ======LOGS====== 660 (527h10m58.618722705s) [noise=true , address=77.90.31.19:4860 , piece_id=5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748001970 (527h10m58.619085005s) [noise=true , address=145.79.227.44:4205 , piece_id=HVNDI6C4SQZYPZIEEP7TC5RW6YNW5WGFDOJKQIIAN3UH3VGDXNIA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748340274 (527h10m58.619423319s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748682386 (527h10m58.619765421s) [noise=true , address=77.90.31.10:4530 , piece_id=K22R6V3CWZWVFKROB23NKYFC5R2Z5LT4KSXMTXFQUUROJKCJWESA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748994651 (527h10m58.620077696s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.749319318 (527h10m58.620402363s) [noise=true , address=145.79.227.40:4060 , piece_id=OADO7RXPLZ4NLGUC2CCXNJQ6OEMU5JWTXCRY7WFWYKKGLMVU6YBA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.749662011 (527h10m58.620745046s) [noise=true , address=77.90.31.12:4601 , piece_id=LCBL5PH44YSNTX6W3UBMGJAGSOTBB6CKMBBD7E3GKDFQBUXMR3AA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.750250261 (527h10m58.621333296s) [noise=true , address=145.79.227.46:4633 , piece_id=U6GO7WKUMICS7QYSJZXZ6ZB33S2MYCV5JGEPVHEG7BHX5HGZL3XA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.750568958 (527h10m58.621651993s) [noise=true , address=145.79.227.23:4579 , piece_id=A3LOK5DMHEU3G3C6XYC2CWLOL7U4XJFXK4RV4A3PH4AXRHTA4EGQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.751689872 (527h10m58.622772917s) [noise=true , address=45.91.116.8:4049 , piece_id=ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.751758763 (527h10m58.622841809s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.753063619 (527h10m58.624146654s) [noise=true , address=145.79.227.36:4537 , piece_id=TMCDI7JYHU6MAPAGTLJTB2GSXMSBPJZJSPZWGAFF5I4VGEGNOMUQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.753568099 (527h10m58.624651134s) [noise=true , address=145.79.227.8:4337 , piece_id=BC5P5Y75VGZDTNDWEOC27UTK3TMWFRT7BNKQTJAYYHPYLYRXV2VQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.754895386 (527h10m58.625978421s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.755999769 (527h10m58.627082804s) [noise=true , address=77.90.31.21:4893 , piece_id=WYF4STSQ27E7KEMRIXKC4ILRHKN4UCNGH5NJFZ5SHOBKW3FNUCHA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.757486991 (527h10m58.628570037s) [noise=true , address=64.49.13.154:4933 , piece_id=V7STA7QTESDS5FDALZIPURQRCO6X4MONA2PTU5RIJHM34P5BQTNA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:40:00.120776880 (527h10m59.991859915s) [upload=462397]: Next returned nil segment Dec 27 14:40:00.120795044 (527h10m59.991878079s) [upload=462397]: Waiting for error group managing segments... Dec 27 14:40:00.475194542 (527h11m0.346277587s) [noise=true , address=64.49.13.154:4933 , piece_id=V7STA7QTESDS5FDALZIPURQRCO6X4MONA2PTU5RIJHM34P5BQTNA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.626897129 (527h11m0.497980174s) [noise=true , address=145.79.227.46:4633 , piece_id=U6GO7WKUMICS7QYSJZXZ6ZB33S2MYCV5JGEPVHEG7BHX5HGZL3XA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.634206858 (527h11m0.505289903s) [noise=true , address=145.79.227.34:4927 , piece_id=3PJB2HW34B5N5LADQYZ3FPNEIWBGKKGJEGADMPJIMWRG6HGQG36Q, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.636936777 (527h11m0.508019822s) [noise=true , address=145.79.227.36:4537 , piece_id=TMCDI7JYHU6MAPAGTLJTB2GSXMSBPJZJSPZWGAFF5I4VGEGNOMUQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.650341221 (527h11m0.521424266s) [noise=true , address=145.79.227.8:4337 , piece_id=BC5P5Y75VGZDTNDWEOC27UTK3TMWFRT7BNKQTJAYYHPYLYRXV2VQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.662509752 (527h11m0.533592797s) [noise=true , address=145.79.227.23:4579 , piece_id=A3LOK5DMHEU3G3C6XYC2CWLOL7U4XJFXK4RV4A3PH4AXRHTA4EGQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.820994414 (527h11m0.692077449s) [noise=true , address=77.90.31.12:4601 , piece_id=LCBL5PH44YSNTX6W3UBMGJAGSOTBB6CKMBBD7E3GKDFQBUXMR3AA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.908728394 (527h11m0.779811429s) [noise=true , address=45.91.116.8:4049 , piece_id=ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.921937597 (527h11m0.793020642s) [noise=true , address=77.90.31.10:4530 , piece_id=K22R6V3CWZWVFKROB23NKYFC5R2Z5LT4KSXMTXFQUUROJKCJWESA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.180318328 (527h11m1.051401374s) [noise=true , address=77.90.31.14:4683 , piece_id=OCZIJHDS4AESHNCHUJIBVN47X336BIKXBLIQ4BVYSF7LAPWIS3KA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.467385073 (527h11m1.338468118s) [noise=true , address=77.90.31.19:4860 , piece_id=5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.506047844 (527h11m1.377130919s) [noise=true , address=145.79.227.44:4205 , piece_id=HVNDI6C4SQZYPZIEEP7TC5RW6YNW5WGFDOJKQIIAN3UH3VGDXNIA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.551036449 (527h11m1.422119494s) [noise=true , address=145.79.227.40:4060 , piece_id=OADO7RXPLZ4NLGUC2CCXNJQ6OEMU5JWTXCRY7WFWYKKGLMVU6YBA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.567791785 (527h11m1.43887482s) [noise=true , address=77.90.31.21:4893 , piece_id=WYF4STSQ27E7KEMRIXKC4ILRHKN4UCNGH5NJFZ5SHOBKW3FNUCHA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.567835187 (527h11m1.438918232s) [seg_pos={0 0}, upload=462397]: Segment reached optimal threshold of 14 pieces. Dec 27 14:40:01.568097526 (527h11m1.439180581s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568109259 (527h11m1.439192304s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568506055 (527h11m1.4395891s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568516865 (527h11m1.43959991s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568673093 (527h11m1.439756148s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection Dec 27 14:40:01.568680948 (527h11m1.439763993s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568844680 (527h11m1.439927725s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FtB4bKTHntdUQxF8G5o9nT73oS2xB7fgD44BkRfKDNFyZnKCk): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568852474 (527h11m1.439935519s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568872633 (527h11m1.439955678s) [seg_pos={0 0}, upload=462397]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 14:40:01.568960319 (527h11m1.440043365s) [upload=462397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:40:01.568966050 (527h11m1.440049095s) [upload=462397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:40:01.862585703 (527h11m1.733668748s) [upload=462397]: Done uploading segments... Dec 27 14:40:01.862613346 (527h11m1.733696401s) [upload=462397]: Upload finished. err: <nil> Duration: 48.80625827s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 7.610817959s trace: https://jaeger.dev.storj.io/trace/2f00e104e333c43f FAIL (2025-12-27 14:45:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== }, upload=462172]: Uploading piece... Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments... Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces. Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments... Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil> Duration: 7.610847847s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [XX.XXXXXXX.XXXXXX.XX.XXXXXX.XXXX.XXXXXXXXXXXX..XXXXXXX.....X] duration: 5.001520877s trace: https://jaeger.dev.storj.io/trace/28b036da00814e3c FAIL (2025-12-27 14:45:48 UTC) orbiter warning: context deadline exceeded Duration: 5.001541836s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X.XXX..X...XX.............X..X....X.........X.X....X...X...] duration: 6.532370616s trace: https://jaeger.dev.storj.io/trace/4f06cb127c622fe FAIL (2025-12-27 14:45:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6163s) [upload=462438]: Uploading segments... Dec 27 14:42:49.799089041 (527h13m49.670172096s) [upload=462438]: Got next segment. Inline: false Dec 27 14:42:49.799144557 (527h13m49.670227602s) [upload=462438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 14:42:49.799150438 (527h13m49.670233483s) [upload=462438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 14:42:50.043758773 (527h13m49.914841808s) [seg_pos={0 0}, upload=462438]: Begin upload segment... Dec 27 14:42:50.043802957 (527h13m49.914885992s) [seg_pos={0 0}, upload=462438]: Done begin upload segment. Dec 27 14:42:50.043842984 (527h13m49.914926029s) [noise=true , address=77.90.31.14:4664 , piece_id=4IP7OZYCQ2LPVW4PVP66PTSKYXYSIXHP25BMWWUSUUEI6SXXMOHQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.044003700 (527h13m49.915086745s) [noise=true , address=145.79.227.50:4375 , piece_id=PO4M5RTMTGXBCVB67CC3NPMYCNI354WGADEJJAIDMBQZZT62JCVQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.044910416 (527h13m49.915993491s) [noise=true , address=45.91.116.6:4975 , piece_id=3HG4RWJFNSJJ3LZDRE5OLAFQILGCPFQXAH4YQCPRBPSGXP7C26GQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.045314165 (527h13m49.9163972s) [noise=true , address=45.91.116.5:4299 , piece_id=YAR5WZGPTKC3UAGTV25YD2LZX3G2CQGWPTYCB3OBJCK2SZEN7MSA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.046014649 (527h13m49.917097684s) [noise=true , address=45.91.116.10:4083 , piece_id=WNEYFFRZ3EQIEP7LQZMAYY3B3KVK7DTYUG6LB4QJ2J5CWGKADAXA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.046429319 (527h13m49.917512364s) [noise=true , address=145.79.227.16:4912 , piece_id=OVJ6A3TWB35UB2KAKLW5MB3DX3VHOWWPF7AMFVD3JVRQPZORGU3Q, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.047612652 (527h13m49.918695697s) [noise=true , address=145.79.227.3:4149 , piece_id=X4KQZTA7BS3HX2HUJCWNUKV4ZC673VTJRZUX6GESZ7M6EKXYTTMQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.049355891 (527h13m49.920438967s) [noise=true , address=145.79.227.29:4941 , piece_id=PR2XNGAX4EMGVZVXZHUMXAMHK7L6TWP6AIO2SABB5JZ5YHVK2LAA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.050241948 (527h13m49.921324993s) [noise=true , address=145.79.227.2:4114 , piece_id=XFVTYCSRMRC5RAQF3X4V6VDHLUDMP7ILK266R36L62DM47PUEHHA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.053153603 (527h13m49.924236648s) [noise=true , address=45.91.116.11:4118 , piece_id=Q3XCMOCYATPS6J6QTUXBQ3ZMVWNR6BDI5NMB7ALKBPZP2V6WKTKQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.053572400 (527h13m49.924655435s) [noise=true , address=77.90.31.10:4517 , piece_id=CJYRTQJM6XQD563UV3WJCI5ZGFUIO2ZAOQ42E4OOAPUBT7GJA3TQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.055566928 (527h13m49.926649963s) [noise=true , address=145.79.227.46:4603 , piece_id=RPYXKRL6XOHOCVZI6HLSFK2ZY3MFGTC3YMPVE3CJZ7QNUIH4KTLQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.056885168 (527h13m49.927968203s) [noise=true , address=45.91.116.8:4048 , piece_id=NPXO5LTRCVZ7KJL42FVT2ZWOSSS7PXAQVJHF7OYPWK4PGPNNGW7A, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.058750811 (527h13m49.929833856s) [noise=true , address=45.91.116.7:4009 , piece_id=34CGGMU5V72HAVH66PKI2A6IYYUGHYDN6CXYBI3GSDJTDHIJBPCA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.060156948 (527h13m49.931239993s) [noise=true , address=145.79.227.4:4186 , piece_id=YRPDW2GS3YDTGUB4VJRWF2S3W7434K4JQWX3UOEAVFR676Z2CDNA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.061310124 (527h13m49.932393169s) [noise=true , address=45.91.116.14:4189 , piece_id=VCTSVCCJ5MHIIDWKHZPWENEEODBT6JMB4FLSWYXSWZIEH6UZV2NA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:51.763514444 (527h13m51.634597489s) [upload=462438]: Next returned nil segment Dec 27 14:42:51.763530965 (527h13m51.634614s) [upload=462438]: Waiting for error group managing segments... Dec 27 14:42:52.048902352 (527h13m51.919985397s) [noise=true , address=145.79.227.16:4912 , piece_id=OVJ6A3TWB35UB2KAKLW5MB3DX3VHOWWPF7AMFVD3JVRQPZORGU3Q, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.063639583 (527h13m51.934722639s) [noise=true , address=145.79.227.4:4186 , piece_id=YRPDW2GS3YDTGUB4VJRWF2S3W7434K4JQWX3UOEAVFR676Z2CDNA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.069432584 (527h13m51.940515619s) [noise=true , address=145.79.227.50:4375 , piece_id=PO4M5RTMTGXBCVB67CC3NPMYCNI354WGADEJJAIDMBQZZT62JCVQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.069628026 (527h13m51.940711072s) [noise=true , address=145.79.227.2:4114 , piece_id=XFVTYCSRMRC5RAQF3X4V6VDHLUDMP7ILK266R36L62DM47PUEHHA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.083496475 (527h13m51.95457953s) [noise=true , address=145.79.227.46:4603 , piece_id=RPYXKRL6XOHOCVZI6HLSFK2ZY3MFGTC3YMPVE3CJZ7QNUIH4KTLQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.086381027 (527h13m51.957464062s) [noise=true , address=145.79.227.3:4149 , piece_id=X4KQZTA7BS3HX2HUJCWNUKV4ZC673VTJRZUX6GESZ7M6EKXYTTMQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.091852015 (527h13m51.96293506s) [noise=true , address=145.79.227.29:4941 , piece_id=PR2XNGAX4EMGVZVXZHUMXAMHK7L6TWP6AIO2SABB5JZ5YHVK2LAA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.254422116 (527h13m52.125505151s) [noise=true , address=77.90.31.10:4517 , piece_id=CJYRTQJM6XQD563UV3WJCI5ZGFUIO2ZAOQ42E4OOAPUBT7GJA3TQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.316266098 (527h13m52.187349133s) [noise=true , address=45.91.116.8:4048 , piece_id=NPXO5LTRCVZ7KJL42FVT2ZWOSSS7PXAQVJHF7OYPWK4PGPNNGW7A, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.324335654 (527h13m52.195418699s) [noise=true , address=45.91.116.10:4083 , piece_id=WNEYFFRZ3EQIEP7LQZMAYY3B3KVK7DTYUG6LB4QJ2J5CWGKADAXA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.456401440 (527h13m52.327484475s) [noise=true , address=45.91.116.14:4189 , piece_id=VCTSVCCJ5MHIIDWKHZPWENEEODBT6JMB4FLSWYXSWZIEH6UZV2NA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.615789081 (527h13m52.486872126s) [noise=true , address=45.91.116.7:4009 , piece_id=34CGGMU5V72HAVH66PKI2A6IYYUGHYDN6CXYBI3GSDJTDHIJBPCA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.936703357 (527h13m52.807786402s) [noise=true , address=77.90.31.14:4664 , piece_id=4IP7OZYCQ2LPVW4PVP66PTSKYXYSIXHP25BMWWUSUUEI6SXXMOHQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.936743273 (527h13m52.807826318s) [seg_pos={0 0}, upload=462438]: Segment reached optimal threshold of 13 pieces. Dec 27 14:42:52.942008298 (527h13m52.813091344s) [noise=true , address=45.91.116.11:4118 , piece_id=Q3XCMOCYATPS6J6QTUXBQ3ZMVWNR6BDI5NMB7ALKBPZP2V6WKTKQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:44872->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:44872->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44872->45.91.116.11:4118: use of closed network connection Dec 27 14:42:52.942024039 (527h13m52.813107074s) [noise=true , address=45.91.116.11:4118 , piece_id=Q3XCMOCYATPS6J6QTUXBQ3ZMVWNR6BDI5NMB7ALKBPZP2V6WKTKQ, seg_pos={0 0}, upload=462438]: Long tail cancellation due to optimal threshold Dec 27 14:42:52.945792765 (527h13m52.81687585s) [noise=true , address=45.91.116.6:4975 , piece_id=3HG4RWJFNSJJ3LZDRE5OLAFQILGCPFQXAH4YQCPRBPSGXP7C26GQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:41890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 14:42:52.945882115 (527h13m52.81696516s) [noise=true , address=45.91.116.6:4975 , piece_id=3HG4RWJFNSJJ3LZDRE5OLAFQILGCPFQXAH4YQCPRBPSGXP7C26GQ, seg_pos={0 0}, upload=462438]: Long tail cancellation due to optimal threshold Dec 27 14:42:52.946209569 (527h13m52.817292624s) [noise=true , address=45.91.116.5:4299 , piece_id=YAR5WZGPTKC3UAGTV25YD2LZX3G2CQGWPTYCB3OBJCK2SZEN7MSA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: read tcp 188.166.225.170:43062->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 14:42:52.946218666 (527h13m52.817301701s) [noise=true , address=45.91.116.5:4299 , piece_id=YAR5WZGPTKC3UAGTV25YD2LZX3G2CQGWPTYCB3OBJCK2SZEN7MSA, seg_pos={0 0}, upload=462438]: Long tail cancellation due to optimal threshold Dec 27 14:42:52.946236721 (527h13m52.817319766s) [seg_pos={0 0}, upload=462438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 14:42:52.946338554 (527h13m52.817421599s) [upload=462438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:42:52.946346790 (527h13m52.817429835s) [upload=462438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:42:53.234888003 (527h13m53.105971048s) [upload=462438]: Done uploading segments... Dec 27 14:42:53.234911567 (527h13m53.105994612s) [upload=462438]: Upload finished. err: <nil> Duration: 6.53239866s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 8.04146857s trace: https://jaeger.dev.storj.io/trace/2f00e104e333c43f FAIL (2025-12-27 14:45:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== }, upload=462172]: Uploading piece... Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments... Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces. Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments... Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil> Duration: 8.041499869s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [XXX.XXXXXXX.XXXXXX.XX.XXXXXX.XXXX.XXXXXXXXXXXX..XXXXXXX.....] duration: 5.00154416s trace: https://jaeger.dev.storj.io/trace/65bb03d629575dbf FAIL (2025-12-27 14:45:11 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001563508s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................X.............X..X...........X.....] duration: 31.668269164s trace: https://jaeger.dev.storj.io/trace/4d753dda46642d0d FAIL (2025-12-27 14:45:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd, Piece ID: 5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA) context deadline exceeded; piecestore: (Node ID: 12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ, Piece ID: ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ) context deadline exceeded; piecestore: (Node ID: 12cth5bdi5iAWhHcLxqN9c7GLdheS7SLQ3x5fEHG1MdgdXWFdD6, Piece ID: OCZIJHDS4AESHNCHUJIBVN47X336BIKXBLIQ4BVYSF7LAPWIS3KA) context deadline exceeded ======LOGS====== 660 (527h10m58.618722705s) [noise=true , address=77.90.31.19:4860 , piece_id=5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748001970 (527h10m58.619085005s) [noise=true , address=145.79.227.44:4205 , piece_id=HVNDI6C4SQZYPZIEEP7TC5RW6YNW5WGFDOJKQIIAN3UH3VGDXNIA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748340274 (527h10m58.619423319s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748682386 (527h10m58.619765421s) [noise=true , address=77.90.31.10:4530 , piece_id=K22R6V3CWZWVFKROB23NKYFC5R2Z5LT4KSXMTXFQUUROJKCJWESA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748994651 (527h10m58.620077696s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.749319318 (527h10m58.620402363s) [noise=true , address=145.79.227.40:4060 , piece_id=OADO7RXPLZ4NLGUC2CCXNJQ6OEMU5JWTXCRY7WFWYKKGLMVU6YBA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.749662011 (527h10m58.620745046s) [noise=true , address=77.90.31.12:4601 , piece_id=LCBL5PH44YSNTX6W3UBMGJAGSOTBB6CKMBBD7E3GKDFQBUXMR3AA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.750250261 (527h10m58.621333296s) [noise=true , address=145.79.227.46:4633 , piece_id=U6GO7WKUMICS7QYSJZXZ6ZB33S2MYCV5JGEPVHEG7BHX5HGZL3XA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.750568958 (527h10m58.621651993s) [noise=true , address=145.79.227.23:4579 , piece_id=A3LOK5DMHEU3G3C6XYC2CWLOL7U4XJFXK4RV4A3PH4AXRHTA4EGQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.751689872 (527h10m58.622772917s) [noise=true , address=45.91.116.8:4049 , piece_id=ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.751758763 (527h10m58.622841809s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.753063619 (527h10m58.624146654s) [noise=true , address=145.79.227.36:4537 , piece_id=TMCDI7JYHU6MAPAGTLJTB2GSXMSBPJZJSPZWGAFF5I4VGEGNOMUQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.753568099 (527h10m58.624651134s) [noise=true , address=145.79.227.8:4337 , piece_id=BC5P5Y75VGZDTNDWEOC27UTK3TMWFRT7BNKQTJAYYHPYLYRXV2VQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.754895386 (527h10m58.625978421s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.755999769 (527h10m58.627082804s) [noise=true , address=77.90.31.21:4893 , piece_id=WYF4STSQ27E7KEMRIXKC4ILRHKN4UCNGH5NJFZ5SHOBKW3FNUCHA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.757486991 (527h10m58.628570037s) [noise=true , address=64.49.13.154:4933 , piece_id=V7STA7QTESDS5FDALZIPURQRCO6X4MONA2PTU5RIJHM34P5BQTNA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:40:00.120776880 (527h10m59.991859915s) [upload=462397]: Next returned nil segment Dec 27 14:40:00.120795044 (527h10m59.991878079s) [upload=462397]: Waiting for error group managing segments... Dec 27 14:40:00.475194542 (527h11m0.346277587s) [noise=true , address=64.49.13.154:4933 , piece_id=V7STA7QTESDS5FDALZIPURQRCO6X4MONA2PTU5RIJHM34P5BQTNA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.626897129 (527h11m0.497980174s) [noise=true , address=145.79.227.46:4633 , piece_id=U6GO7WKUMICS7QYSJZXZ6ZB33S2MYCV5JGEPVHEG7BHX5HGZL3XA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.634206858 (527h11m0.505289903s) [noise=true , address=145.79.227.34:4927 , piece_id=3PJB2HW34B5N5LADQYZ3FPNEIWBGKKGJEGADMPJIMWRG6HGQG36Q, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.636936777 (527h11m0.508019822s) [noise=true , address=145.79.227.36:4537 , piece_id=TMCDI7JYHU6MAPAGTLJTB2GSXMSBPJZJSPZWGAFF5I4VGEGNOMUQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.650341221 (527h11m0.521424266s) [noise=true , address=145.79.227.8:4337 , piece_id=BC5P5Y75VGZDTNDWEOC27UTK3TMWFRT7BNKQTJAYYHPYLYRXV2VQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.662509752 (527h11m0.533592797s) [noise=true , address=145.79.227.23:4579 , piece_id=A3LOK5DMHEU3G3C6XYC2CWLOL7U4XJFXK4RV4A3PH4AXRHTA4EGQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.820994414 (527h11m0.692077449s) [noise=true , address=77.90.31.12:4601 , piece_id=LCBL5PH44YSNTX6W3UBMGJAGSOTBB6CKMBBD7E3GKDFQBUXMR3AA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.908728394 (527h11m0.779811429s) [noise=true , address=45.91.116.8:4049 , piece_id=ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.921937597 (527h11m0.793020642s) [noise=true , address=77.90.31.10:4530 , piece_id=K22R6V3CWZWVFKROB23NKYFC5R2Z5LT4KSXMTXFQUUROJKCJWESA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.180318328 (527h11m1.051401374s) [noise=true , address=77.90.31.14:4683 , piece_id=OCZIJHDS4AESHNCHUJIBVN47X336BIKXBLIQ4BVYSF7LAPWIS3KA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.467385073 (527h11m1.338468118s) [noise=true , address=77.90.31.19:4860 , piece_id=5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.506047844 (527h11m1.377130919s) [noise=true , address=145.79.227.44:4205 , piece_id=HVNDI6C4SQZYPZIEEP7TC5RW6YNW5WGFDOJKQIIAN3UH3VGDXNIA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.551036449 (527h11m1.422119494s) [noise=true , address=145.79.227.40:4060 , piece_id=OADO7RXPLZ4NLGUC2CCXNJQ6OEMU5JWTXCRY7WFWYKKGLMVU6YBA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.567791785 (527h11m1.43887482s) [noise=true , address=77.90.31.21:4893 , piece_id=WYF4STSQ27E7KEMRIXKC4ILRHKN4UCNGH5NJFZ5SHOBKW3FNUCHA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.567835187 (527h11m1.438918232s) [seg_pos={0 0}, upload=462397]: Segment reached optimal threshold of 14 pieces. Dec 27 14:40:01.568097526 (527h11m1.439180581s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568109259 (527h11m1.439192304s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568506055 (527h11m1.4395891s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568516865 (527h11m1.43959991s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568673093 (527h11m1.439756148s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection Dec 27 14:40:01.568680948 (527h11m1.439763993s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568844680 (527h11m1.439927725s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FtB4bKTHntdUQxF8G5o9nT73oS2xB7fgD44BkRfKDNFyZnKCk): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568852474 (527h11m1.439935519s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568872633 (527h11m1.439955678s) [seg_pos={0 0}, upload=462397]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 14:40:01.568960319 (527h11m1.440043365s) [upload=462397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:40:01.568966050 (527h11m1.440049095s) [upload=462397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:40:01.862585703 (527h11m1.733668748s) [upload=462397]: Done uploading segments... Dec 27 14:40:01.862613346 (527h11m1.733696401s) [upload=462397]: Upload finished. err: <nil> Duration: 31.668290285s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 10.733991642s trace: https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a FAIL (2025-12-27 14:45:07 UTC) orbiter warning: context deadline exceeded Duration: 10.734019878s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X.XXX..X...XX.............X..X....X.........X.X....X...X..] duration: 5.645195562s trace: https://jaeger.dev.storj.io/trace/4f06cb127c622fe FAIL (2025-12-27 14:44:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6163s) [upload=462438]: Uploading segments... Dec 27 14:42:49.799089041 (527h13m49.670172096s) [upload=462438]: Got next segment. Inline: false Dec 27 14:42:49.799144557 (527h13m49.670227602s) [upload=462438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 14:42:49.799150438 (527h13m49.670233483s) [upload=462438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 14:42:50.043758773 (527h13m49.914841808s) [seg_pos={0 0}, upload=462438]: Begin upload segment... Dec 27 14:42:50.043802957 (527h13m49.914885992s) [seg_pos={0 0}, upload=462438]: Done begin upload segment. Dec 27 14:42:50.043842984 (527h13m49.914926029s) [noise=true , address=77.90.31.14:4664 , piece_id=4IP7OZYCQ2LPVW4PVP66PTSKYXYSIXHP25BMWWUSUUEI6SXXMOHQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.044003700 (527h13m49.915086745s) [noise=true , address=145.79.227.50:4375 , piece_id=PO4M5RTMTGXBCVB67CC3NPMYCNI354WGADEJJAIDMBQZZT62JCVQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.044910416 (527h13m49.915993491s) [noise=true , address=45.91.116.6:4975 , piece_id=3HG4RWJFNSJJ3LZDRE5OLAFQILGCPFQXAH4YQCPRBPSGXP7C26GQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.045314165 (527h13m49.9163972s) [noise=true , address=45.91.116.5:4299 , piece_id=YAR5WZGPTKC3UAGTV25YD2LZX3G2CQGWPTYCB3OBJCK2SZEN7MSA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.046014649 (527h13m49.917097684s) [noise=true , address=45.91.116.10:4083 , piece_id=WNEYFFRZ3EQIEP7LQZMAYY3B3KVK7DTYUG6LB4QJ2J5CWGKADAXA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.046429319 (527h13m49.917512364s) [noise=true , address=145.79.227.16:4912 , piece_id=OVJ6A3TWB35UB2KAKLW5MB3DX3VHOWWPF7AMFVD3JVRQPZORGU3Q, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.047612652 (527h13m49.918695697s) [noise=true , address=145.79.227.3:4149 , piece_id=X4KQZTA7BS3HX2HUJCWNUKV4ZC673VTJRZUX6GESZ7M6EKXYTTMQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.049355891 (527h13m49.920438967s) [noise=true , address=145.79.227.29:4941 , piece_id=PR2XNGAX4EMGVZVXZHUMXAMHK7L6TWP6AIO2SABB5JZ5YHVK2LAA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.050241948 (527h13m49.921324993s) [noise=true , address=145.79.227.2:4114 , piece_id=XFVTYCSRMRC5RAQF3X4V6VDHLUDMP7ILK266R36L62DM47PUEHHA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.053153603 (527h13m49.924236648s) [noise=true , address=45.91.116.11:4118 , piece_id=Q3XCMOCYATPS6J6QTUXBQ3ZMVWNR6BDI5NMB7ALKBPZP2V6WKTKQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.053572400 (527h13m49.924655435s) [noise=true , address=77.90.31.10:4517 , piece_id=CJYRTQJM6XQD563UV3WJCI5ZGFUIO2ZAOQ42E4OOAPUBT7GJA3TQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.055566928 (527h13m49.926649963s) [noise=true , address=145.79.227.46:4603 , piece_id=RPYXKRL6XOHOCVZI6HLSFK2ZY3MFGTC3YMPVE3CJZ7QNUIH4KTLQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.056885168 (527h13m49.927968203s) [noise=true , address=45.91.116.8:4048 , piece_id=NPXO5LTRCVZ7KJL42FVT2ZWOSSS7PXAQVJHF7OYPWK4PGPNNGW7A, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.058750811 (527h13m49.929833856s) [noise=true , address=45.91.116.7:4009 , piece_id=34CGGMU5V72HAVH66PKI2A6IYYUGHYDN6CXYBI3GSDJTDHIJBPCA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.060156948 (527h13m49.931239993s) [noise=true , address=145.79.227.4:4186 , piece_id=YRPDW2GS3YDTGUB4VJRWF2S3W7434K4JQWX3UOEAVFR676Z2CDNA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.061310124 (527h13m49.932393169s) [noise=true , address=45.91.116.14:4189 , piece_id=VCTSVCCJ5MHIIDWKHZPWENEEODBT6JMB4FLSWYXSWZIEH6UZV2NA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:51.763514444 (527h13m51.634597489s) [upload=462438]: Next returned nil segment Dec 27 14:42:51.763530965 (527h13m51.634614s) [upload=462438]: Waiting for error group managing segments... Dec 27 14:42:52.048902352 (527h13m51.919985397s) [noise=true , address=145.79.227.16:4912 , piece_id=OVJ6A3TWB35UB2KAKLW5MB3DX3VHOWWPF7AMFVD3JVRQPZORGU3Q, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.063639583 (527h13m51.934722639s) [noise=true , address=145.79.227.4:4186 , piece_id=YRPDW2GS3YDTGUB4VJRWF2S3W7434K4JQWX3UOEAVFR676Z2CDNA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.069432584 (527h13m51.940515619s) [noise=true , address=145.79.227.50:4375 , piece_id=PO4M5RTMTGXBCVB67CC3NPMYCNI354WGADEJJAIDMBQZZT62JCVQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.069628026 (527h13m51.940711072s) [noise=true , address=145.79.227.2:4114 , piece_id=XFVTYCSRMRC5RAQF3X4V6VDHLUDMP7ILK266R36L62DM47PUEHHA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.083496475 (527h13m51.95457953s) [noise=true , address=145.79.227.46:4603 , piece_id=RPYXKRL6XOHOCVZI6HLSFK2ZY3MFGTC3YMPVE3CJZ7QNUIH4KTLQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.086381027 (527h13m51.957464062s) [noise=true , address=145.79.227.3:4149 , piece_id=X4KQZTA7BS3HX2HUJCWNUKV4ZC673VTJRZUX6GESZ7M6EKXYTTMQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.091852015 (527h13m51.96293506s) [noise=true , address=145.79.227.29:4941 , piece_id=PR2XNGAX4EMGVZVXZHUMXAMHK7L6TWP6AIO2SABB5JZ5YHVK2LAA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.254422116 (527h13m52.125505151s) [noise=true , address=77.90.31.10:4517 , piece_id=CJYRTQJM6XQD563UV3WJCI5ZGFUIO2ZAOQ42E4OOAPUBT7GJA3TQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.316266098 (527h13m52.187349133s) [noise=true , address=45.91.116.8:4048 , piece_id=NPXO5LTRCVZ7KJL42FVT2ZWOSSS7PXAQVJHF7OYPWK4PGPNNGW7A, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.324335654 (527h13m52.195418699s) [noise=true , address=45.91.116.10:4083 , piece_id=WNEYFFRZ3EQIEP7LQZMAYY3B3KVK7DTYUG6LB4QJ2J5CWGKADAXA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.456401440 (527h13m52.327484475s) [noise=true , address=45.91.116.14:4189 , piece_id=VCTSVCCJ5MHIIDWKHZPWENEEODBT6JMB4FLSWYXSWZIEH6UZV2NA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.615789081 (527h13m52.486872126s) [noise=true , address=45.91.116.7:4009 , piece_id=34CGGMU5V72HAVH66PKI2A6IYYUGHYDN6CXYBI3GSDJTDHIJBPCA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.936703357 (527h13m52.807786402s) [noise=true , address=77.90.31.14:4664 , piece_id=4IP7OZYCQ2LPVW4PVP66PTSKYXYSIXHP25BMWWUSUUEI6SXXMOHQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.936743273 (527h13m52.807826318s) [seg_pos={0 0}, upload=462438]: Segment reached optimal threshold of 13 pieces. Dec 27 14:42:52.942008298 (527h13m52.813091344s) [noise=true , address=45.91.116.11:4118 , piece_id=Q3XCMOCYATPS6J6QTUXBQ3ZMVWNR6BDI5NMB7ALKBPZP2V6WKTKQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:44872->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:44872->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44872->45.91.116.11:4118: use of closed network connection Dec 27 14:42:52.942024039 (527h13m52.813107074s) [noise=true , address=45.91.116.11:4118 , piece_id=Q3XCMOCYATPS6J6QTUXBQ3ZMVWNR6BDI5NMB7ALKBPZP2V6WKTKQ, seg_pos={0 0}, upload=462438]: Long tail cancellation due to optimal threshold Dec 27 14:42:52.945792765 (527h13m52.81687585s) [noise=true , address=45.91.116.6:4975 , piece_id=3HG4RWJFNSJJ3LZDRE5OLAFQILGCPFQXAH4YQCPRBPSGXP7C26GQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:41890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 14:42:52.945882115 (527h13m52.81696516s) [noise=true , address=45.91.116.6:4975 , piece_id=3HG4RWJFNSJJ3LZDRE5OLAFQILGCPFQXAH4YQCPRBPSGXP7C26GQ, seg_pos={0 0}, upload=462438]: Long tail cancellation due to optimal threshold Dec 27 14:42:52.946209569 (527h13m52.817292624s) [noise=true , address=45.91.116.5:4299 , piece_id=YAR5WZGPTKC3UAGTV25YD2LZX3G2CQGWPTYCB3OBJCK2SZEN7MSA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: read tcp 188.166.225.170:43062->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 14:42:52.946218666 (527h13m52.817301701s) [noise=true , address=45.91.116.5:4299 , piece_id=YAR5WZGPTKC3UAGTV25YD2LZX3G2CQGWPTYCB3OBJCK2SZEN7MSA, seg_pos={0 0}, upload=462438]: Long tail cancellation due to optimal threshold Dec 27 14:42:52.946236721 (527h13m52.817319766s) [seg_pos={0 0}, upload=462438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 14:42:52.946338554 (527h13m52.817421599s) [upload=462438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:42:52.946346790 (527h13m52.817429835s) [upload=462438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:42:53.234888003 (527h13m53.105971048s) [upload=462438]: Done uploading segments... Dec 27 14:42:53.234911567 (527h13m53.105994612s) [upload=462438]: Upload finished. err: <nil> Duration: 5.64523134s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................X.............X..X...........X....] duration: 55.89164786s trace: https://jaeger.dev.storj.io/trace/4d753dda46642d0d FAIL (2025-12-27 14:44:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd, Piece ID: 5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA) context deadline exceeded; piecestore: (Node ID: 12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ, Piece ID: ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ) context deadline exceeded; piecestore: (Node ID: 12cth5bdi5iAWhHcLxqN9c7GLdheS7SLQ3x5fEHG1MdgdXWFdD6, Piece ID: OCZIJHDS4AESHNCHUJIBVN47X336BIKXBLIQ4BVYSF7LAPWIS3KA) context deadline exceeded ======LOGS====== 660 (527h10m58.618722705s) [noise=true , address=77.90.31.19:4860 , piece_id=5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748001970 (527h10m58.619085005s) [noise=true , address=145.79.227.44:4205 , piece_id=HVNDI6C4SQZYPZIEEP7TC5RW6YNW5WGFDOJKQIIAN3UH3VGDXNIA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748340274 (527h10m58.619423319s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748682386 (527h10m58.619765421s) [noise=true , address=77.90.31.10:4530 , piece_id=K22R6V3CWZWVFKROB23NKYFC5R2Z5LT4KSXMTXFQUUROJKCJWESA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748994651 (527h10m58.620077696s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.749319318 (527h10m58.620402363s) [noise=true , address=145.79.227.40:4060 , piece_id=OADO7RXPLZ4NLGUC2CCXNJQ6OEMU5JWTXCRY7WFWYKKGLMVU6YBA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.749662011 (527h10m58.620745046s) [noise=true , address=77.90.31.12:4601 , piece_id=LCBL5PH44YSNTX6W3UBMGJAGSOTBB6CKMBBD7E3GKDFQBUXMR3AA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.750250261 (527h10m58.621333296s) [noise=true , address=145.79.227.46:4633 , piece_id=U6GO7WKUMICS7QYSJZXZ6ZB33S2MYCV5JGEPVHEG7BHX5HGZL3XA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.750568958 (527h10m58.621651993s) [noise=true , address=145.79.227.23:4579 , piece_id=A3LOK5DMHEU3G3C6XYC2CWLOL7U4XJFXK4RV4A3PH4AXRHTA4EGQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.751689872 (527h10m58.622772917s) [noise=true , address=45.91.116.8:4049 , piece_id=ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.751758763 (527h10m58.622841809s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.753063619 (527h10m58.624146654s) [noise=true , address=145.79.227.36:4537 , piece_id=TMCDI7JYHU6MAPAGTLJTB2GSXMSBPJZJSPZWGAFF5I4VGEGNOMUQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.753568099 (527h10m58.624651134s) [noise=true , address=145.79.227.8:4337 , piece_id=BC5P5Y75VGZDTNDWEOC27UTK3TMWFRT7BNKQTJAYYHPYLYRXV2VQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.754895386 (527h10m58.625978421s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.755999769 (527h10m58.627082804s) [noise=true , address=77.90.31.21:4893 , piece_id=WYF4STSQ27E7KEMRIXKC4ILRHKN4UCNGH5NJFZ5SHOBKW3FNUCHA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.757486991 (527h10m58.628570037s) [noise=true , address=64.49.13.154:4933 , piece_id=V7STA7QTESDS5FDALZIPURQRCO6X4MONA2PTU5RIJHM34P5BQTNA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:40:00.120776880 (527h10m59.991859915s) [upload=462397]: Next returned nil segment Dec 27 14:40:00.120795044 (527h10m59.991878079s) [upload=462397]: Waiting for error group managing segments... Dec 27 14:40:00.475194542 (527h11m0.346277587s) [noise=true , address=64.49.13.154:4933 , piece_id=V7STA7QTESDS5FDALZIPURQRCO6X4MONA2PTU5RIJHM34P5BQTNA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.626897129 (527h11m0.497980174s) [noise=true , address=145.79.227.46:4633 , piece_id=U6GO7WKUMICS7QYSJZXZ6ZB33S2MYCV5JGEPVHEG7BHX5HGZL3XA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.634206858 (527h11m0.505289903s) [noise=true , address=145.79.227.34:4927 , piece_id=3PJB2HW34B5N5LADQYZ3FPNEIWBGKKGJEGADMPJIMWRG6HGQG36Q, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.636936777 (527h11m0.508019822s) [noise=true , address=145.79.227.36:4537 , piece_id=TMCDI7JYHU6MAPAGTLJTB2GSXMSBPJZJSPZWGAFF5I4VGEGNOMUQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.650341221 (527h11m0.521424266s) [noise=true , address=145.79.227.8:4337 , piece_id=BC5P5Y75VGZDTNDWEOC27UTK3TMWFRT7BNKQTJAYYHPYLYRXV2VQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.662509752 (527h11m0.533592797s) [noise=true , address=145.79.227.23:4579 , piece_id=A3LOK5DMHEU3G3C6XYC2CWLOL7U4XJFXK4RV4A3PH4AXRHTA4EGQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.820994414 (527h11m0.692077449s) [noise=true , address=77.90.31.12:4601 , piece_id=LCBL5PH44YSNTX6W3UBMGJAGSOTBB6CKMBBD7E3GKDFQBUXMR3AA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.908728394 (527h11m0.779811429s) [noise=true , address=45.91.116.8:4049 , piece_id=ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.921937597 (527h11m0.793020642s) [noise=true , address=77.90.31.10:4530 , piece_id=K22R6V3CWZWVFKROB23NKYFC5R2Z5LT4KSXMTXFQUUROJKCJWESA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.180318328 (527h11m1.051401374s) [noise=true , address=77.90.31.14:4683 , piece_id=OCZIJHDS4AESHNCHUJIBVN47X336BIKXBLIQ4BVYSF7LAPWIS3KA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.467385073 (527h11m1.338468118s) [noise=true , address=77.90.31.19:4860 , piece_id=5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.506047844 (527h11m1.377130919s) [noise=true , address=145.79.227.44:4205 , piece_id=HVNDI6C4SQZYPZIEEP7TC5RW6YNW5WGFDOJKQIIAN3UH3VGDXNIA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.551036449 (527h11m1.422119494s) [noise=true , address=145.79.227.40:4060 , piece_id=OADO7RXPLZ4NLGUC2CCXNJQ6OEMU5JWTXCRY7WFWYKKGLMVU6YBA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.567791785 (527h11m1.43887482s) [noise=true , address=77.90.31.21:4893 , piece_id=WYF4STSQ27E7KEMRIXKC4ILRHKN4UCNGH5NJFZ5SHOBKW3FNUCHA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.567835187 (527h11m1.438918232s) [seg_pos={0 0}, upload=462397]: Segment reached optimal threshold of 14 pieces. Dec 27 14:40:01.568097526 (527h11m1.439180581s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568109259 (527h11m1.439192304s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568506055 (527h11m1.4395891s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568516865 (527h11m1.43959991s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568673093 (527h11m1.439756148s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection Dec 27 14:40:01.568680948 (527h11m1.439763993s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568844680 (527h11m1.439927725s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FtB4bKTHntdUQxF8G5o9nT73oS2xB7fgD44BkRfKDNFyZnKCk): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568852474 (527h11m1.439935519s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568872633 (527h11m1.439955678s) [seg_pos={0 0}, upload=462397]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 14:40:01.568960319 (527h11m1.440043365s) [upload=462397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:40:01.568966050 (527h11m1.440049095s) [upload=462397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:40:01.862585703 (527h11m1.733668748s) [upload=462397]: Done uploading segments... Dec 27 14:40:01.862613346 (527h11m1.733696401s) [upload=462397]: Upload finished. err: <nil> Duration: 55.891664632s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 14.117859982s trace: https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a FAIL (2025-12-27 14:44:25 UTC) orbiter warning: context deadline exceeded Duration: 14.11788557s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 9.368953287s trace: https://jaeger.dev.storj.io/trace/2f00e104e333c43f FAIL (2025-12-27 14:44:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== }, upload=462172]: Uploading piece... Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments... Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces. Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments... Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil> Duration: 9.368989175s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [XXXX.XXXXXXX.XXXXXX.XX.XXXXXX.XXXX.XXXXXXXXXXXX..XXXXXXX....] duration: 5.000685074s trace: https://jaeger.dev.storj.io/trace/65bb03d629575dbf FAIL (2025-12-27 14:44:07 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000715942s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................X.............X..X...........X...] duration: 43.715615615s trace: https://jaeger.dev.storj.io/trace/4d753dda46642d0d FAIL (2025-12-27 14:43:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd, Piece ID: 5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA) context deadline exceeded; piecestore: (Node ID: 12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ, Piece ID: ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ) context deadline exceeded; piecestore: (Node ID: 12cth5bdi5iAWhHcLxqN9c7GLdheS7SLQ3x5fEHG1MdgdXWFdD6, Piece ID: OCZIJHDS4AESHNCHUJIBVN47X336BIKXBLIQ4BVYSF7LAPWIS3KA) context deadline exceeded ======LOGS====== 660 (527h10m58.618722705s) [noise=true , address=77.90.31.19:4860 , piece_id=5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748001970 (527h10m58.619085005s) [noise=true , address=145.79.227.44:4205 , piece_id=HVNDI6C4SQZYPZIEEP7TC5RW6YNW5WGFDOJKQIIAN3UH3VGDXNIA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748340274 (527h10m58.619423319s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748682386 (527h10m58.619765421s) [noise=true , address=77.90.31.10:4530 , piece_id=K22R6V3CWZWVFKROB23NKYFC5R2Z5LT4KSXMTXFQUUROJKCJWESA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748994651 (527h10m58.620077696s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.749319318 (527h10m58.620402363s) [noise=true , address=145.79.227.40:4060 , piece_id=OADO7RXPLZ4NLGUC2CCXNJQ6OEMU5JWTXCRY7WFWYKKGLMVU6YBA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.749662011 (527h10m58.620745046s) [noise=true , address=77.90.31.12:4601 , piece_id=LCBL5PH44YSNTX6W3UBMGJAGSOTBB6CKMBBD7E3GKDFQBUXMR3AA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.750250261 (527h10m58.621333296s) [noise=true , address=145.79.227.46:4633 , piece_id=U6GO7WKUMICS7QYSJZXZ6ZB33S2MYCV5JGEPVHEG7BHX5HGZL3XA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.750568958 (527h10m58.621651993s) [noise=true , address=145.79.227.23:4579 , piece_id=A3LOK5DMHEU3G3C6XYC2CWLOL7U4XJFXK4RV4A3PH4AXRHTA4EGQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.751689872 (527h10m58.622772917s) [noise=true , address=45.91.116.8:4049 , piece_id=ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.751758763 (527h10m58.622841809s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.753063619 (527h10m58.624146654s) [noise=true , address=145.79.227.36:4537 , piece_id=TMCDI7JYHU6MAPAGTLJTB2GSXMSBPJZJSPZWGAFF5I4VGEGNOMUQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.753568099 (527h10m58.624651134s) [noise=true , address=145.79.227.8:4337 , piece_id=BC5P5Y75VGZDTNDWEOC27UTK3TMWFRT7BNKQTJAYYHPYLYRXV2VQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.754895386 (527h10m58.625978421s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.755999769 (527h10m58.627082804s) [noise=true , address=77.90.31.21:4893 , piece_id=WYF4STSQ27E7KEMRIXKC4ILRHKN4UCNGH5NJFZ5SHOBKW3FNUCHA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.757486991 (527h10m58.628570037s) [noise=true , address=64.49.13.154:4933 , piece_id=V7STA7QTESDS5FDALZIPURQRCO6X4MONA2PTU5RIJHM34P5BQTNA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:40:00.120776880 (527h10m59.991859915s) [upload=462397]: Next returned nil segment Dec 27 14:40:00.120795044 (527h10m59.991878079s) [upload=462397]: Waiting for error group managing segments... Dec 27 14:40:00.475194542 (527h11m0.346277587s) [noise=true , address=64.49.13.154:4933 , piece_id=V7STA7QTESDS5FDALZIPURQRCO6X4MONA2PTU5RIJHM34P5BQTNA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.626897129 (527h11m0.497980174s) [noise=true , address=145.79.227.46:4633 , piece_id=U6GO7WKUMICS7QYSJZXZ6ZB33S2MYCV5JGEPVHEG7BHX5HGZL3XA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.634206858 (527h11m0.505289903s) [noise=true , address=145.79.227.34:4927 , piece_id=3PJB2HW34B5N5LADQYZ3FPNEIWBGKKGJEGADMPJIMWRG6HGQG36Q, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.636936777 (527h11m0.508019822s) [noise=true , address=145.79.227.36:4537 , piece_id=TMCDI7JYHU6MAPAGTLJTB2GSXMSBPJZJSPZWGAFF5I4VGEGNOMUQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.650341221 (527h11m0.521424266s) [noise=true , address=145.79.227.8:4337 , piece_id=BC5P5Y75VGZDTNDWEOC27UTK3TMWFRT7BNKQTJAYYHPYLYRXV2VQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.662509752 (527h11m0.533592797s) [noise=true , address=145.79.227.23:4579 , piece_id=A3LOK5DMHEU3G3C6XYC2CWLOL7U4XJFXK4RV4A3PH4AXRHTA4EGQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.820994414 (527h11m0.692077449s) [noise=true , address=77.90.31.12:4601 , piece_id=LCBL5PH44YSNTX6W3UBMGJAGSOTBB6CKMBBD7E3GKDFQBUXMR3AA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.908728394 (527h11m0.779811429s) [noise=true , address=45.91.116.8:4049 , piece_id=ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.921937597 (527h11m0.793020642s) [noise=true , address=77.90.31.10:4530 , piece_id=K22R6V3CWZWVFKROB23NKYFC5R2Z5LT4KSXMTXFQUUROJKCJWESA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.180318328 (527h11m1.051401374s) [noise=true , address=77.90.31.14:4683 , piece_id=OCZIJHDS4AESHNCHUJIBVN47X336BIKXBLIQ4BVYSF7LAPWIS3KA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.467385073 (527h11m1.338468118s) [noise=true , address=77.90.31.19:4860 , piece_id=5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.506047844 (527h11m1.377130919s) [noise=true , address=145.79.227.44:4205 , piece_id=HVNDI6C4SQZYPZIEEP7TC5RW6YNW5WGFDOJKQIIAN3UH3VGDXNIA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.551036449 (527h11m1.422119494s) [noise=true , address=145.79.227.40:4060 , piece_id=OADO7RXPLZ4NLGUC2CCXNJQ6OEMU5JWTXCRY7WFWYKKGLMVU6YBA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.567791785 (527h11m1.43887482s) [noise=true , address=77.90.31.21:4893 , piece_id=WYF4STSQ27E7KEMRIXKC4ILRHKN4UCNGH5NJFZ5SHOBKW3FNUCHA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.567835187 (527h11m1.438918232s) [seg_pos={0 0}, upload=462397]: Segment reached optimal threshold of 14 pieces. Dec 27 14:40:01.568097526 (527h11m1.439180581s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568109259 (527h11m1.439192304s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568506055 (527h11m1.4395891s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568516865 (527h11m1.43959991s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568673093 (527h11m1.439756148s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection Dec 27 14:40:01.568680948 (527h11m1.439763993s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568844680 (527h11m1.439927725s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FtB4bKTHntdUQxF8G5o9nT73oS2xB7fgD44BkRfKDNFyZnKCk): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568852474 (527h11m1.439935519s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568872633 (527h11m1.439955678s) [seg_pos={0 0}, upload=462397]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 14:40:01.568960319 (527h11m1.440043365s) [upload=462397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:40:01.568966050 (527h11m1.440049095s) [upload=462397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:40:01.862585703 (527h11m1.733668748s) [upload=462397]: Done uploading segments... Dec 27 14:40:01.862613346 (527h11m1.733696401s) [upload=462397]: Upload finished. err: <nil> Duration: 43.715643658s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [.XXXX.XXXXXXX.XXXXXX.XX.XXXXXX.XXXX.XXXXXXXXXXXX..XXXXXXX...] duration: 5.001310076s trace: https://jaeger.dev.storj.io/trace/65bb03d629575dbf FAIL (2025-12-27 14:43:39 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001329512s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.801881603s trace: https://jaeger.dev.storj.io/trace/2f00e104e333c43f FAIL (2025-12-27 14:43:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== }, upload=462172]: Uploading piece... Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments... Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces. Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments... Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil> Duration: 6.801910276s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...X.XXX..X...XX.............X..X....X.........X.X....X...X.] duration: 6.722285588s trace: https://jaeger.dev.storj.io/trace/4f06cb127c622fe FAIL (2025-12-27 14:43:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6163s) [upload=462438]: Uploading segments... Dec 27 14:42:49.799089041 (527h13m49.670172096s) [upload=462438]: Got next segment. Inline: false Dec 27 14:42:49.799144557 (527h13m49.670227602s) [upload=462438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 14:42:49.799150438 (527h13m49.670233483s) [upload=462438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 14:42:50.043758773 (527h13m49.914841808s) [seg_pos={0 0}, upload=462438]: Begin upload segment... Dec 27 14:42:50.043802957 (527h13m49.914885992s) [seg_pos={0 0}, upload=462438]: Done begin upload segment. Dec 27 14:42:50.043842984 (527h13m49.914926029s) [noise=true , address=77.90.31.14:4664 , piece_id=4IP7OZYCQ2LPVW4PVP66PTSKYXYSIXHP25BMWWUSUUEI6SXXMOHQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.044003700 (527h13m49.915086745s) [noise=true , address=145.79.227.50:4375 , piece_id=PO4M5RTMTGXBCVB67CC3NPMYCNI354WGADEJJAIDMBQZZT62JCVQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.044910416 (527h13m49.915993491s) [noise=true , address=45.91.116.6:4975 , piece_id=3HG4RWJFNSJJ3LZDRE5OLAFQILGCPFQXAH4YQCPRBPSGXP7C26GQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.045314165 (527h13m49.9163972s) [noise=true , address=45.91.116.5:4299 , piece_id=YAR5WZGPTKC3UAGTV25YD2LZX3G2CQGWPTYCB3OBJCK2SZEN7MSA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.046014649 (527h13m49.917097684s) [noise=true , address=45.91.116.10:4083 , piece_id=WNEYFFRZ3EQIEP7LQZMAYY3B3KVK7DTYUG6LB4QJ2J5CWGKADAXA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.046429319 (527h13m49.917512364s) [noise=true , address=145.79.227.16:4912 , piece_id=OVJ6A3TWB35UB2KAKLW5MB3DX3VHOWWPF7AMFVD3JVRQPZORGU3Q, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.047612652 (527h13m49.918695697s) [noise=true , address=145.79.227.3:4149 , piece_id=X4KQZTA7BS3HX2HUJCWNUKV4ZC673VTJRZUX6GESZ7M6EKXYTTMQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.049355891 (527h13m49.920438967s) [noise=true , address=145.79.227.29:4941 , piece_id=PR2XNGAX4EMGVZVXZHUMXAMHK7L6TWP6AIO2SABB5JZ5YHVK2LAA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.050241948 (527h13m49.921324993s) [noise=true , address=145.79.227.2:4114 , piece_id=XFVTYCSRMRC5RAQF3X4V6VDHLUDMP7ILK266R36L62DM47PUEHHA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.053153603 (527h13m49.924236648s) [noise=true , address=45.91.116.11:4118 , piece_id=Q3XCMOCYATPS6J6QTUXBQ3ZMVWNR6BDI5NMB7ALKBPZP2V6WKTKQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.053572400 (527h13m49.924655435s) [noise=true , address=77.90.31.10:4517 , piece_id=CJYRTQJM6XQD563UV3WJCI5ZGFUIO2ZAOQ42E4OOAPUBT7GJA3TQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.055566928 (527h13m49.926649963s) [noise=true , address=145.79.227.46:4603 , piece_id=RPYXKRL6XOHOCVZI6HLSFK2ZY3MFGTC3YMPVE3CJZ7QNUIH4KTLQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.056885168 (527h13m49.927968203s) [noise=true , address=45.91.116.8:4048 , piece_id=NPXO5LTRCVZ7KJL42FVT2ZWOSSS7PXAQVJHF7OYPWK4PGPNNGW7A, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.058750811 (527h13m49.929833856s) [noise=true , address=45.91.116.7:4009 , piece_id=34CGGMU5V72HAVH66PKI2A6IYYUGHYDN6CXYBI3GSDJTDHIJBPCA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.060156948 (527h13m49.931239993s) [noise=true , address=145.79.227.4:4186 , piece_id=YRPDW2GS3YDTGUB4VJRWF2S3W7434K4JQWX3UOEAVFR676Z2CDNA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.061310124 (527h13m49.932393169s) [noise=true , address=45.91.116.14:4189 , piece_id=VCTSVCCJ5MHIIDWKHZPWENEEODBT6JMB4FLSWYXSWZIEH6UZV2NA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:51.763514444 (527h13m51.634597489s) [upload=462438]: Next returned nil segment Dec 27 14:42:51.763530965 (527h13m51.634614s) [upload=462438]: Waiting for error group managing segments... Dec 27 14:42:52.048902352 (527h13m51.919985397s) [noise=true , address=145.79.227.16:4912 , piece_id=OVJ6A3TWB35UB2KAKLW5MB3DX3VHOWWPF7AMFVD3JVRQPZORGU3Q, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.063639583 (527h13m51.934722639s) [noise=true , address=145.79.227.4:4186 , piece_id=YRPDW2GS3YDTGUB4VJRWF2S3W7434K4JQWX3UOEAVFR676Z2CDNA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.069432584 (527h13m51.940515619s) [noise=true , address=145.79.227.50:4375 , piece_id=PO4M5RTMTGXBCVB67CC3NPMYCNI354WGADEJJAIDMBQZZT62JCVQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.069628026 (527h13m51.940711072s) [noise=true , address=145.79.227.2:4114 , piece_id=XFVTYCSRMRC5RAQF3X4V6VDHLUDMP7ILK266R36L62DM47PUEHHA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.083496475 (527h13m51.95457953s) [noise=true , address=145.79.227.46:4603 , piece_id=RPYXKRL6XOHOCVZI6HLSFK2ZY3MFGTC3YMPVE3CJZ7QNUIH4KTLQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.086381027 (527h13m51.957464062s) [noise=true , address=145.79.227.3:4149 , piece_id=X4KQZTA7BS3HX2HUJCWNUKV4ZC673VTJRZUX6GESZ7M6EKXYTTMQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.091852015 (527h13m51.96293506s) [noise=true , address=145.79.227.29:4941 , piece_id=PR2XNGAX4EMGVZVXZHUMXAMHK7L6TWP6AIO2SABB5JZ5YHVK2LAA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.254422116 (527h13m52.125505151s) [noise=true , address=77.90.31.10:4517 , piece_id=CJYRTQJM6XQD563UV3WJCI5ZGFUIO2ZAOQ42E4OOAPUBT7GJA3TQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.316266098 (527h13m52.187349133s) [noise=true , address=45.91.116.8:4048 , piece_id=NPXO5LTRCVZ7KJL42FVT2ZWOSSS7PXAQVJHF7OYPWK4PGPNNGW7A, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.324335654 (527h13m52.195418699s) [noise=true , address=45.91.116.10:4083 , piece_id=WNEYFFRZ3EQIEP7LQZMAYY3B3KVK7DTYUG6LB4QJ2J5CWGKADAXA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.456401440 (527h13m52.327484475s) [noise=true , address=45.91.116.14:4189 , piece_id=VCTSVCCJ5MHIIDWKHZPWENEEODBT6JMB4FLSWYXSWZIEH6UZV2NA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.615789081 (527h13m52.486872126s) [noise=true , address=45.91.116.7:4009 , piece_id=34CGGMU5V72HAVH66PKI2A6IYYUGHYDN6CXYBI3GSDJTDHIJBPCA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.936703357 (527h13m52.807786402s) [noise=true , address=77.90.31.14:4664 , piece_id=4IP7OZYCQ2LPVW4PVP66PTSKYXYSIXHP25BMWWUSUUEI6SXXMOHQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.936743273 (527h13m52.807826318s) [seg_pos={0 0}, upload=462438]: Segment reached optimal threshold of 13 pieces. Dec 27 14:42:52.942008298 (527h13m52.813091344s) [noise=true , address=45.91.116.11:4118 , piece_id=Q3XCMOCYATPS6J6QTUXBQ3ZMVWNR6BDI5NMB7ALKBPZP2V6WKTKQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:44872->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:44872->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44872->45.91.116.11:4118: use of closed network connection Dec 27 14:42:52.942024039 (527h13m52.813107074s) [noise=true , address=45.91.116.11:4118 , piece_id=Q3XCMOCYATPS6J6QTUXBQ3ZMVWNR6BDI5NMB7ALKBPZP2V6WKTKQ, seg_pos={0 0}, upload=462438]: Long tail cancellation due to optimal threshold Dec 27 14:42:52.945792765 (527h13m52.81687585s) [noise=true , address=45.91.116.6:4975 , piece_id=3HG4RWJFNSJJ3LZDRE5OLAFQILGCPFQXAH4YQCPRBPSGXP7C26GQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:41890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 14:42:52.945882115 (527h13m52.81696516s) [noise=true , address=45.91.116.6:4975 , piece_id=3HG4RWJFNSJJ3LZDRE5OLAFQILGCPFQXAH4YQCPRBPSGXP7C26GQ, seg_pos={0 0}, upload=462438]: Long tail cancellation due to optimal threshold Dec 27 14:42:52.946209569 (527h13m52.817292624s) [noise=true , address=45.91.116.5:4299 , piece_id=YAR5WZGPTKC3UAGTV25YD2LZX3G2CQGWPTYCB3OBJCK2SZEN7MSA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: read tcp 188.166.225.170:43062->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 14:42:52.946218666 (527h13m52.817301701s) [noise=true , address=45.91.116.5:4299 , piece_id=YAR5WZGPTKC3UAGTV25YD2LZX3G2CQGWPTYCB3OBJCK2SZEN7MSA, seg_pos={0 0}, upload=462438]: Long tail cancellation due to optimal threshold Dec 27 14:42:52.946236721 (527h13m52.817319766s) [seg_pos={0 0}, upload=462438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 14:42:52.946338554 (527h13m52.817421599s) [upload=462438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:42:52.946346790 (527h13m52.817429835s) [upload=462438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:42:53.234888003 (527h13m53.105971048s) [upload=462438]: Done uploading segments... Dec 27 14:42:53.234911567 (527h13m53.105994612s) [upload=462438]: Upload finished. err: <nil> Duration: 6.72231229s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.XX.........................................................] duration: 7.504388806s trace: https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a FAIL (2025-12-27 14:43:23 UTC) orbiter warning: context deadline exceeded Duration: 7.504422569s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [..XXXX.XXXXXXX.XXXXXX.XX.XXXXXX.XXXX.XXXXXXXXXXXX..XXXXXXX..] duration: 5.000927525s trace: https://jaeger.dev.storj.io/trace/65bb03d629575dbf FAIL (2025-12-27 14:43:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00095107s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [....X.XXX..X...XX.............X..X....X.........X.X....X...X] duration: 30.002714804s trace: https://jaeger.dev.storj.io/trace/4f06cb127c622fe FAIL (2025-12-27 14:43:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6163s) [upload=462438]: Uploading segments... Dec 27 14:42:49.799089041 (527h13m49.670172096s) [upload=462438]: Got next segment. Inline: false Dec 27 14:42:49.799144557 (527h13m49.670227602s) [upload=462438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 14:42:49.799150438 (527h13m49.670233483s) [upload=462438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 14:42:50.043758773 (527h13m49.914841808s) [seg_pos={0 0}, upload=462438]: Begin upload segment... Dec 27 14:42:50.043802957 (527h13m49.914885992s) [seg_pos={0 0}, upload=462438]: Done begin upload segment. Dec 27 14:42:50.043842984 (527h13m49.914926029s) [noise=true , address=77.90.31.14:4664 , piece_id=4IP7OZYCQ2LPVW4PVP66PTSKYXYSIXHP25BMWWUSUUEI6SXXMOHQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.044003700 (527h13m49.915086745s) [noise=true , address=145.79.227.50:4375 , piece_id=PO4M5RTMTGXBCVB67CC3NPMYCNI354WGADEJJAIDMBQZZT62JCVQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.044910416 (527h13m49.915993491s) [noise=true , address=45.91.116.6:4975 , piece_id=3HG4RWJFNSJJ3LZDRE5OLAFQILGCPFQXAH4YQCPRBPSGXP7C26GQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.045314165 (527h13m49.9163972s) [noise=true , address=45.91.116.5:4299 , piece_id=YAR5WZGPTKC3UAGTV25YD2LZX3G2CQGWPTYCB3OBJCK2SZEN7MSA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.046014649 (527h13m49.917097684s) [noise=true , address=45.91.116.10:4083 , piece_id=WNEYFFRZ3EQIEP7LQZMAYY3B3KVK7DTYUG6LB4QJ2J5CWGKADAXA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.046429319 (527h13m49.917512364s) [noise=true , address=145.79.227.16:4912 , piece_id=OVJ6A3TWB35UB2KAKLW5MB3DX3VHOWWPF7AMFVD3JVRQPZORGU3Q, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.047612652 (527h13m49.918695697s) [noise=true , address=145.79.227.3:4149 , piece_id=X4KQZTA7BS3HX2HUJCWNUKV4ZC673VTJRZUX6GESZ7M6EKXYTTMQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.049355891 (527h13m49.920438967s) [noise=true , address=145.79.227.29:4941 , piece_id=PR2XNGAX4EMGVZVXZHUMXAMHK7L6TWP6AIO2SABB5JZ5YHVK2LAA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.050241948 (527h13m49.921324993s) [noise=true , address=145.79.227.2:4114 , piece_id=XFVTYCSRMRC5RAQF3X4V6VDHLUDMP7ILK266R36L62DM47PUEHHA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.053153603 (527h13m49.924236648s) [noise=true , address=45.91.116.11:4118 , piece_id=Q3XCMOCYATPS6J6QTUXBQ3ZMVWNR6BDI5NMB7ALKBPZP2V6WKTKQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.053572400 (527h13m49.924655435s) [noise=true , address=77.90.31.10:4517 , piece_id=CJYRTQJM6XQD563UV3WJCI5ZGFUIO2ZAOQ42E4OOAPUBT7GJA3TQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.055566928 (527h13m49.926649963s) [noise=true , address=145.79.227.46:4603 , piece_id=RPYXKRL6XOHOCVZI6HLSFK2ZY3MFGTC3YMPVE3CJZ7QNUIH4KTLQ, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.056885168 (527h13m49.927968203s) [noise=true , address=45.91.116.8:4048 , piece_id=NPXO5LTRCVZ7KJL42FVT2ZWOSSS7PXAQVJHF7OYPWK4PGPNNGW7A, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.058750811 (527h13m49.929833856s) [noise=true , address=45.91.116.7:4009 , piece_id=34CGGMU5V72HAVH66PKI2A6IYYUGHYDN6CXYBI3GSDJTDHIJBPCA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.060156948 (527h13m49.931239993s) [noise=true , address=145.79.227.4:4186 , piece_id=YRPDW2GS3YDTGUB4VJRWF2S3W7434K4JQWX3UOEAVFR676Z2CDNA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:50.061310124 (527h13m49.932393169s) [noise=true , address=45.91.116.14:4189 , piece_id=VCTSVCCJ5MHIIDWKHZPWENEEODBT6JMB4FLSWYXSWZIEH6UZV2NA, seg_pos={0 0}, upload=462438]: Uploading piece... Dec 27 14:42:51.763514444 (527h13m51.634597489s) [upload=462438]: Next returned nil segment Dec 27 14:42:51.763530965 (527h13m51.634614s) [upload=462438]: Waiting for error group managing segments... Dec 27 14:42:52.048902352 (527h13m51.919985397s) [noise=true , address=145.79.227.16:4912 , piece_id=OVJ6A3TWB35UB2KAKLW5MB3DX3VHOWWPF7AMFVD3JVRQPZORGU3Q, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.063639583 (527h13m51.934722639s) [noise=true , address=145.79.227.4:4186 , piece_id=YRPDW2GS3YDTGUB4VJRWF2S3W7434K4JQWX3UOEAVFR676Z2CDNA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.069432584 (527h13m51.940515619s) [noise=true , address=145.79.227.50:4375 , piece_id=PO4M5RTMTGXBCVB67CC3NPMYCNI354WGADEJJAIDMBQZZT62JCVQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.069628026 (527h13m51.940711072s) [noise=true , address=145.79.227.2:4114 , piece_id=XFVTYCSRMRC5RAQF3X4V6VDHLUDMP7ILK266R36L62DM47PUEHHA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.083496475 (527h13m51.95457953s) [noise=true , address=145.79.227.46:4603 , piece_id=RPYXKRL6XOHOCVZI6HLSFK2ZY3MFGTC3YMPVE3CJZ7QNUIH4KTLQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.086381027 (527h13m51.957464062s) [noise=true , address=145.79.227.3:4149 , piece_id=X4KQZTA7BS3HX2HUJCWNUKV4ZC673VTJRZUX6GESZ7M6EKXYTTMQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.091852015 (527h13m51.96293506s) [noise=true , address=145.79.227.29:4941 , piece_id=PR2XNGAX4EMGVZVXZHUMXAMHK7L6TWP6AIO2SABB5JZ5YHVK2LAA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.254422116 (527h13m52.125505151s) [noise=true , address=77.90.31.10:4517 , piece_id=CJYRTQJM6XQD563UV3WJCI5ZGFUIO2ZAOQ42E4OOAPUBT7GJA3TQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.316266098 (527h13m52.187349133s) [noise=true , address=45.91.116.8:4048 , piece_id=NPXO5LTRCVZ7KJL42FVT2ZWOSSS7PXAQVJHF7OYPWK4PGPNNGW7A, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.324335654 (527h13m52.195418699s) [noise=true , address=45.91.116.10:4083 , piece_id=WNEYFFRZ3EQIEP7LQZMAYY3B3KVK7DTYUG6LB4QJ2J5CWGKADAXA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.456401440 (527h13m52.327484475s) [noise=true , address=45.91.116.14:4189 , piece_id=VCTSVCCJ5MHIIDWKHZPWENEEODBT6JMB4FLSWYXSWZIEH6UZV2NA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.615789081 (527h13m52.486872126s) [noise=true , address=45.91.116.7:4009 , piece_id=34CGGMU5V72HAVH66PKI2A6IYYUGHYDN6CXYBI3GSDJTDHIJBPCA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.936703357 (527h13m52.807786402s) [noise=true , address=77.90.31.14:4664 , piece_id=4IP7OZYCQ2LPVW4PVP66PTSKYXYSIXHP25BMWWUSUUEI6SXXMOHQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: <nil> Dec 27 14:42:52.936743273 (527h13m52.807826318s) [seg_pos={0 0}, upload=462438]: Segment reached optimal threshold of 13 pieces. Dec 27 14:42:52.942008298 (527h13m52.813091344s) [noise=true , address=45.91.116.11:4118 , piece_id=Q3XCMOCYATPS6J6QTUXBQ3ZMVWNR6BDI5NMB7ALKBPZP2V6WKTKQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:44872->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:44872->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44872->45.91.116.11:4118: use of closed network connection Dec 27 14:42:52.942024039 (527h13m52.813107074s) [noise=true , address=45.91.116.11:4118 , piece_id=Q3XCMOCYATPS6J6QTUXBQ3ZMVWNR6BDI5NMB7ALKBPZP2V6WKTKQ, seg_pos={0 0}, upload=462438]: Long tail cancellation due to optimal threshold Dec 27 14:42:52.945792765 (527h13m52.81687585s) [noise=true , address=45.91.116.6:4975 , piece_id=3HG4RWJFNSJJ3LZDRE5OLAFQILGCPFQXAH4YQCPRBPSGXP7C26GQ, seg_pos={0 0}, upload=462438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:41890->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 14:42:52.945882115 (527h13m52.81696516s) [noise=true , address=45.91.116.6:4975 , piece_id=3HG4RWJFNSJJ3LZDRE5OLAFQILGCPFQXAH4YQCPRBPSGXP7C26GQ, seg_pos={0 0}, upload=462438]: Long tail cancellation due to optimal threshold Dec 27 14:42:52.946209569 (527h13m52.817292624s) [noise=true , address=45.91.116.5:4299 , piece_id=YAR5WZGPTKC3UAGTV25YD2LZX3G2CQGWPTYCB3OBJCK2SZEN7MSA, seg_pos={0 0}, upload=462438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: read tcp 188.166.225.170:43062->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 14:42:52.946218666 (527h13m52.817301701s) [noise=true , address=45.91.116.5:4299 , piece_id=YAR5WZGPTKC3UAGTV25YD2LZX3G2CQGWPTYCB3OBJCK2SZEN7MSA, seg_pos={0 0}, upload=462438]: Long tail cancellation due to optimal threshold Dec 27 14:42:52.946236721 (527h13m52.817319766s) [seg_pos={0 0}, upload=462438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 14:42:52.946338554 (527h13m52.817421599s) [upload=462438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:42:52.946346790 (527h13m52.817429835s) [upload=462438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:42:53.234888003 (527h13m53.105971048s) [upload=462438]: Done uploading segments... Dec 27 14:42:53.234911567 (527h13m53.105994612s) [upload=462438]: Upload finished. err: <nil> Duration: 30.002737406s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..XX........................................................] duration: 7.557842861s trace: https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a FAIL (2025-12-27 14:42:57 UTC) orbiter warning: context deadline exceeded Duration: 7.557857159s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [X..XXXX.XXXXXXX.XXXXXX.XX.XXXXXX.XXXX.XXXXXXXXXXXX..XXXXXXX.] duration: 4.836398022s trace: https://jaeger.dev.storj.io/trace/65bb03d629575dbf FAIL (2025-12-27 14:42:56 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.836428601s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................X.............X..X...........X..] duration: 53.458853889s trace: https://jaeger.dev.storj.io/trace/4d753dda46642d0d FAIL (2025-12-27 14:42:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd, Piece ID: 5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA) context deadline exceeded; piecestore: (Node ID: 12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ, Piece ID: ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ) context deadline exceeded; piecestore: (Node ID: 12cth5bdi5iAWhHcLxqN9c7GLdheS7SLQ3x5fEHG1MdgdXWFdD6, Piece ID: OCZIJHDS4AESHNCHUJIBVN47X336BIKXBLIQ4BVYSF7LAPWIS3KA) context deadline exceeded ======LOGS====== 660 (527h10m58.618722705s) [noise=true , address=77.90.31.19:4860 , piece_id=5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748001970 (527h10m58.619085005s) [noise=true , address=145.79.227.44:4205 , piece_id=HVNDI6C4SQZYPZIEEP7TC5RW6YNW5WGFDOJKQIIAN3UH3VGDXNIA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748340274 (527h10m58.619423319s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748682386 (527h10m58.619765421s) [noise=true , address=77.90.31.10:4530 , piece_id=K22R6V3CWZWVFKROB23NKYFC5R2Z5LT4KSXMTXFQUUROJKCJWESA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748994651 (527h10m58.620077696s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.749319318 (527h10m58.620402363s) [noise=true , address=145.79.227.40:4060 , piece_id=OADO7RXPLZ4NLGUC2CCXNJQ6OEMU5JWTXCRY7WFWYKKGLMVU6YBA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.749662011 (527h10m58.620745046s) [noise=true , address=77.90.31.12:4601 , piece_id=LCBL5PH44YSNTX6W3UBMGJAGSOTBB6CKMBBD7E3GKDFQBUXMR3AA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.750250261 (527h10m58.621333296s) [noise=true , address=145.79.227.46:4633 , piece_id=U6GO7WKUMICS7QYSJZXZ6ZB33S2MYCV5JGEPVHEG7BHX5HGZL3XA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.750568958 (527h10m58.621651993s) [noise=true , address=145.79.227.23:4579 , piece_id=A3LOK5DMHEU3G3C6XYC2CWLOL7U4XJFXK4RV4A3PH4AXRHTA4EGQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.751689872 (527h10m58.622772917s) [noise=true , address=45.91.116.8:4049 , piece_id=ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.751758763 (527h10m58.622841809s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.753063619 (527h10m58.624146654s) [noise=true , address=145.79.227.36:4537 , piece_id=TMCDI7JYHU6MAPAGTLJTB2GSXMSBPJZJSPZWGAFF5I4VGEGNOMUQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.753568099 (527h10m58.624651134s) [noise=true , address=145.79.227.8:4337 , piece_id=BC5P5Y75VGZDTNDWEOC27UTK3TMWFRT7BNKQTJAYYHPYLYRXV2VQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.754895386 (527h10m58.625978421s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.755999769 (527h10m58.627082804s) [noise=true , address=77.90.31.21:4893 , piece_id=WYF4STSQ27E7KEMRIXKC4ILRHKN4UCNGH5NJFZ5SHOBKW3FNUCHA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.757486991 (527h10m58.628570037s) [noise=true , address=64.49.13.154:4933 , piece_id=V7STA7QTESDS5FDALZIPURQRCO6X4MONA2PTU5RIJHM34P5BQTNA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:40:00.120776880 (527h10m59.991859915s) [upload=462397]: Next returned nil segment Dec 27 14:40:00.120795044 (527h10m59.991878079s) [upload=462397]: Waiting for error group managing segments... Dec 27 14:40:00.475194542 (527h11m0.346277587s) [noise=true , address=64.49.13.154:4933 , piece_id=V7STA7QTESDS5FDALZIPURQRCO6X4MONA2PTU5RIJHM34P5BQTNA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.626897129 (527h11m0.497980174s) [noise=true , address=145.79.227.46:4633 , piece_id=U6GO7WKUMICS7QYSJZXZ6ZB33S2MYCV5JGEPVHEG7BHX5HGZL3XA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.634206858 (527h11m0.505289903s) [noise=true , address=145.79.227.34:4927 , piece_id=3PJB2HW34B5N5LADQYZ3FPNEIWBGKKGJEGADMPJIMWRG6HGQG36Q, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.636936777 (527h11m0.508019822s) [noise=true , address=145.79.227.36:4537 , piece_id=TMCDI7JYHU6MAPAGTLJTB2GSXMSBPJZJSPZWGAFF5I4VGEGNOMUQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.650341221 (527h11m0.521424266s) [noise=true , address=145.79.227.8:4337 , piece_id=BC5P5Y75VGZDTNDWEOC27UTK3TMWFRT7BNKQTJAYYHPYLYRXV2VQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.662509752 (527h11m0.533592797s) [noise=true , address=145.79.227.23:4579 , piece_id=A3LOK5DMHEU3G3C6XYC2CWLOL7U4XJFXK4RV4A3PH4AXRHTA4EGQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.820994414 (527h11m0.692077449s) [noise=true , address=77.90.31.12:4601 , piece_id=LCBL5PH44YSNTX6W3UBMGJAGSOTBB6CKMBBD7E3GKDFQBUXMR3AA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.908728394 (527h11m0.779811429s) [noise=true , address=45.91.116.8:4049 , piece_id=ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.921937597 (527h11m0.793020642s) [noise=true , address=77.90.31.10:4530 , piece_id=K22R6V3CWZWVFKROB23NKYFC5R2Z5LT4KSXMTXFQUUROJKCJWESA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.180318328 (527h11m1.051401374s) [noise=true , address=77.90.31.14:4683 , piece_id=OCZIJHDS4AESHNCHUJIBVN47X336BIKXBLIQ4BVYSF7LAPWIS3KA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.467385073 (527h11m1.338468118s) [noise=true , address=77.90.31.19:4860 , piece_id=5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.506047844 (527h11m1.377130919s) [noise=true , address=145.79.227.44:4205 , piece_id=HVNDI6C4SQZYPZIEEP7TC5RW6YNW5WGFDOJKQIIAN3UH3VGDXNIA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.551036449 (527h11m1.422119494s) [noise=true , address=145.79.227.40:4060 , piece_id=OADO7RXPLZ4NLGUC2CCXNJQ6OEMU5JWTXCRY7WFWYKKGLMVU6YBA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.567791785 (527h11m1.43887482s) [noise=true , address=77.90.31.21:4893 , piece_id=WYF4STSQ27E7KEMRIXKC4ILRHKN4UCNGH5NJFZ5SHOBKW3FNUCHA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.567835187 (527h11m1.438918232s) [seg_pos={0 0}, upload=462397]: Segment reached optimal threshold of 14 pieces. Dec 27 14:40:01.568097526 (527h11m1.439180581s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568109259 (527h11m1.439192304s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568506055 (527h11m1.4395891s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568516865 (527h11m1.43959991s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568673093 (527h11m1.439756148s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection Dec 27 14:40:01.568680948 (527h11m1.439763993s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568844680 (527h11m1.439927725s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FtB4bKTHntdUQxF8G5o9nT73oS2xB7fgD44BkRfKDNFyZnKCk): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568852474 (527h11m1.439935519s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568872633 (527h11m1.439955678s) [seg_pos={0 0}, upload=462397]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 14:40:01.568960319 (527h11m1.440043365s) [upload=462397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:40:01.568966050 (527h11m1.440049095s) [upload=462397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:40:01.862585703 (527h11m1.733668748s) [upload=462397]: Done uploading segments... Dec 27 14:40:01.862613346 (527h11m1.733696401s) [upload=462397]: Upload finished. err: <nil> Duration: 53.458872826s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [.X..XXXX.XXXXXXX.XXXXXX.XX.XXXXXX.XXXX.XXXXXXXXXXXX..XXXXXXX] duration: 5.000928125s trace: https://jaeger.dev.storj.io/trace/65bb03d629575dbf FAIL (2025-12-27 14:42:37 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000966208s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 7.312965073s trace: https://jaeger.dev.storj.io/trace/2f00e104e333c43f FAIL (2025-12-27 14:42:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== }, upload=462172]: Uploading piece... Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments... Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces. Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments... Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil> Duration: 7.312994288s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...XX.......................................................] duration: 7.343586529s trace: https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a FAIL (2025-12-27 14:42:09 UTC) orbiter warning: context deadline exceeded Duration: 7.343619282s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [X.X..XXXX.XXXXXXX.XXXXXX.XX.XXXXXX.XXXX.XXXXXXXXXXXX..XXXXXX] duration: 5.002065742s trace: https://jaeger.dev.storj.io/trace/ac4a88e35c97561 FAIL (2025-12-27 14:42:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002092995s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................X.............X..X...........X.] duration: 45.782449248s trace: https://jaeger.dev.storj.io/trace/4d753dda46642d0d FAIL (2025-12-27 14:41:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd, Piece ID: 5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA) context deadline exceeded; piecestore: (Node ID: 12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ, Piece ID: ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ) context deadline exceeded; piecestore: (Node ID: 12cth5bdi5iAWhHcLxqN9c7GLdheS7SLQ3x5fEHG1MdgdXWFdD6, Piece ID: OCZIJHDS4AESHNCHUJIBVN47X336BIKXBLIQ4BVYSF7LAPWIS3KA) context deadline exceeded ======LOGS====== 660 (527h10m58.618722705s) [noise=true , address=77.90.31.19:4860 , piece_id=5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748001970 (527h10m58.619085005s) [noise=true , address=145.79.227.44:4205 , piece_id=HVNDI6C4SQZYPZIEEP7TC5RW6YNW5WGFDOJKQIIAN3UH3VGDXNIA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748340274 (527h10m58.619423319s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748682386 (527h10m58.619765421s) [noise=true , address=77.90.31.10:4530 , piece_id=K22R6V3CWZWVFKROB23NKYFC5R2Z5LT4KSXMTXFQUUROJKCJWESA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748994651 (527h10m58.620077696s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.749319318 (527h10m58.620402363s) [noise=true , address=145.79.227.40:4060 , piece_id=OADO7RXPLZ4NLGUC2CCXNJQ6OEMU5JWTXCRY7WFWYKKGLMVU6YBA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.749662011 (527h10m58.620745046s) [noise=true , address=77.90.31.12:4601 , piece_id=LCBL5PH44YSNTX6W3UBMGJAGSOTBB6CKMBBD7E3GKDFQBUXMR3AA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.750250261 (527h10m58.621333296s) [noise=true , address=145.79.227.46:4633 , piece_id=U6GO7WKUMICS7QYSJZXZ6ZB33S2MYCV5JGEPVHEG7BHX5HGZL3XA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.750568958 (527h10m58.621651993s) [noise=true , address=145.79.227.23:4579 , piece_id=A3LOK5DMHEU3G3C6XYC2CWLOL7U4XJFXK4RV4A3PH4AXRHTA4EGQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.751689872 (527h10m58.622772917s) [noise=true , address=45.91.116.8:4049 , piece_id=ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.751758763 (527h10m58.622841809s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.753063619 (527h10m58.624146654s) [noise=true , address=145.79.227.36:4537 , piece_id=TMCDI7JYHU6MAPAGTLJTB2GSXMSBPJZJSPZWGAFF5I4VGEGNOMUQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.753568099 (527h10m58.624651134s) [noise=true , address=145.79.227.8:4337 , piece_id=BC5P5Y75VGZDTNDWEOC27UTK3TMWFRT7BNKQTJAYYHPYLYRXV2VQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.754895386 (527h10m58.625978421s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.755999769 (527h10m58.627082804s) [noise=true , address=77.90.31.21:4893 , piece_id=WYF4STSQ27E7KEMRIXKC4ILRHKN4UCNGH5NJFZ5SHOBKW3FNUCHA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.757486991 (527h10m58.628570037s) [noise=true , address=64.49.13.154:4933 , piece_id=V7STA7QTESDS5FDALZIPURQRCO6X4MONA2PTU5RIJHM34P5BQTNA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:40:00.120776880 (527h10m59.991859915s) [upload=462397]: Next returned nil segment Dec 27 14:40:00.120795044 (527h10m59.991878079s) [upload=462397]: Waiting for error group managing segments... Dec 27 14:40:00.475194542 (527h11m0.346277587s) [noise=true , address=64.49.13.154:4933 , piece_id=V7STA7QTESDS5FDALZIPURQRCO6X4MONA2PTU5RIJHM34P5BQTNA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.626897129 (527h11m0.497980174s) [noise=true , address=145.79.227.46:4633 , piece_id=U6GO7WKUMICS7QYSJZXZ6ZB33S2MYCV5JGEPVHEG7BHX5HGZL3XA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.634206858 (527h11m0.505289903s) [noise=true , address=145.79.227.34:4927 , piece_id=3PJB2HW34B5N5LADQYZ3FPNEIWBGKKGJEGADMPJIMWRG6HGQG36Q, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.636936777 (527h11m0.508019822s) [noise=true , address=145.79.227.36:4537 , piece_id=TMCDI7JYHU6MAPAGTLJTB2GSXMSBPJZJSPZWGAFF5I4VGEGNOMUQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.650341221 (527h11m0.521424266s) [noise=true , address=145.79.227.8:4337 , piece_id=BC5P5Y75VGZDTNDWEOC27UTK3TMWFRT7BNKQTJAYYHPYLYRXV2VQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.662509752 (527h11m0.533592797s) [noise=true , address=145.79.227.23:4579 , piece_id=A3LOK5DMHEU3G3C6XYC2CWLOL7U4XJFXK4RV4A3PH4AXRHTA4EGQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.820994414 (527h11m0.692077449s) [noise=true , address=77.90.31.12:4601 , piece_id=LCBL5PH44YSNTX6W3UBMGJAGSOTBB6CKMBBD7E3GKDFQBUXMR3AA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.908728394 (527h11m0.779811429s) [noise=true , address=45.91.116.8:4049 , piece_id=ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.921937597 (527h11m0.793020642s) [noise=true , address=77.90.31.10:4530 , piece_id=K22R6V3CWZWVFKROB23NKYFC5R2Z5LT4KSXMTXFQUUROJKCJWESA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.180318328 (527h11m1.051401374s) [noise=true , address=77.90.31.14:4683 , piece_id=OCZIJHDS4AESHNCHUJIBVN47X336BIKXBLIQ4BVYSF7LAPWIS3KA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.467385073 (527h11m1.338468118s) [noise=true , address=77.90.31.19:4860 , piece_id=5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.506047844 (527h11m1.377130919s) [noise=true , address=145.79.227.44:4205 , piece_id=HVNDI6C4SQZYPZIEEP7TC5RW6YNW5WGFDOJKQIIAN3UH3VGDXNIA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.551036449 (527h11m1.422119494s) [noise=true , address=145.79.227.40:4060 , piece_id=OADO7RXPLZ4NLGUC2CCXNJQ6OEMU5JWTXCRY7WFWYKKGLMVU6YBA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.567791785 (527h11m1.43887482s) [noise=true , address=77.90.31.21:4893 , piece_id=WYF4STSQ27E7KEMRIXKC4ILRHKN4UCNGH5NJFZ5SHOBKW3FNUCHA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.567835187 (527h11m1.438918232s) [seg_pos={0 0}, upload=462397]: Segment reached optimal threshold of 14 pieces. Dec 27 14:40:01.568097526 (527h11m1.439180581s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568109259 (527h11m1.439192304s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568506055 (527h11m1.4395891s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568516865 (527h11m1.43959991s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568673093 (527h11m1.439756148s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection Dec 27 14:40:01.568680948 (527h11m1.439763993s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568844680 (527h11m1.439927725s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FtB4bKTHntdUQxF8G5o9nT73oS2xB7fgD44BkRfKDNFyZnKCk): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568852474 (527h11m1.439935519s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568872633 (527h11m1.439955678s) [seg_pos={0 0}, upload=462397]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 14:40:01.568960319 (527h11m1.440043365s) [upload=462397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:40:01.568966050 (527h11m1.440049095s) [upload=462397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:40:01.862585703 (527h11m1.733668748s) [upload=462397]: Done uploading segments... Dec 27 14:40:01.862613346 (527h11m1.733696401s) [upload=462397]: Upload finished. err: <nil> Duration: 45.782464878s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.....X.XXX..X...XX.............X..X....X.........X.X....X...] duration: 6.017339582s trace: https://jaeger.dev.storj.io/trace/4770d9925deace45 FAIL (2025-12-27 14:41:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 14:39:34.904922529 (527h10m34.776005564s) [upload=462392]: Starting upload Dec 27 14:39:34.905040364 (527h10m34.776123399s) [upload=462392]: Done starting upload Dec 27 14:39:34.905154030 (527h10m34.776237075s) [upload=462392]: Uploading segments... Dec 27 14:39:34.905261495 (527h10m34.77634454s) [upload=462392]: Got next segment. Inline: false Dec 27 14:39:34.905309938 (527h10m34.776392973s) [upload=462392]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 14:39:34.905314837 (527h10m34.776397872s) [upload=462392]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 14:39:35.151760912 (527h10m35.022843957s) [seg_pos={0 0}, upload=462392]: Begin upload segment... Dec 27 14:39:35.151814034 (527h10m35.022897079s) [seg_pos={0 0}, upload=462392]: Done begin upload segment. Dec 27 14:39:35.151865551 (527h10m35.022948586s) [noise=true , address=77.90.31.10:4520 , piece_id=TU7IIIV4Q3MYLNEUHGPOVKFU5POHKS57HM5PLYXTVYV4H2544XRQ, seg_pos={0 0}, upload=462392]: Uploading piece... Dec 27 14:39:35.151871462 (527h10m35.022954497s) [noise=true , address=145.79.227.35:4481 , piece_id=HTNUI4NFDQJU7BVCGXPBRMMOXG3U55RHXHGZVIJOEWJ7GQSJ2CZA, seg_pos={0 0}, upload=462392]: Uploading piece... Dec 27 14:39:35.151969700 (527h10m35.023052745s) [noise=true , address=145.79.227.42:4123 , piece_id=6NOG42HTTE5MUIM3RD47DPA7LGDIQ3E4MGF5NR3LS2HQOU4MIEEA, seg_pos={0 0}, upload=462392]: Uploading piece... Dec 27 14:39:35.152456136 (527h10m35.023539171s) [noise=true , address=145.79.227.2:4113 , piece_id=EFTQU5HHSHQS6HPHOSMIVVUIHDS62IOG65MLN6WJEN2SNSXQFLWA, seg_pos={0 0}, upload=462392]: Uploading piece... Dec 27 14:39:35.152806964 (527h10m35.023889999s) [noise=true , address=45.91.116.6:4976 , piece_id=J4TMBLY35XHJZB7GZYCLGLFRTHPSTIYK4QBHBQMBZ7KYY7Q5AVQQ, seg_pos={0 0}, upload=462392]: Uploading piece... Dec 27 14:39:35.153114930 (527h10m35.024197975s) [noise=true , address=45.91.116.7:4010 , piece_id=MDERFXEOP7WCC7NPECA6QNLZSVTF3ZBCVXMKIS7DDNRV3TZRCCCA, seg_pos={0 0}, upload=462392]: Uploading piece... Dec 27 14:39:35.153415894 (527h10m35.024498929s) [noise=true , address=145.79.227.26:4655 , piece_id=L7223FKI5SQTGYT7SYQNSGCWOXHBAK7HEBCABPTQWW7NSG2FHH6A, seg_pos={0 0}, upload=462392]: Uploading piece... Dec 27 14:39:35.153966112 (527h10m35.025049157s) [noise=true , address=45.91.116.4:4940 , piece_id=H2U4QZL7BTK7VWRY4KYLHINSW325Q2N4GPWCTMH2KNOCRIVZ7VJQ, seg_pos={0 0}, upload=462392]: Uploading piece... Dec 27 14:39:35.154432469 (527h10m35.025515514s) [noise=true , address=145.79.227.45:4231 , piece_id=LVR4D2J2SNKCTGIUKMMIFE45SKSKCTRFXFWQ5MABGLXILJUN6UIQ, seg_pos={0 0}, upload=462392]: Uploading piece... Dec 27 14:39:35.154903005 (527h10m35.02598605s) [noise=true , address=145.79.227.46:4603 , piece_id=73NJC7PAUTDZST3I6HS44TXZHDE7JYOQICVQSRMQTGQMBQMPYGZQ, seg_pos={0 0}, upload=462392]: Uploading piece... Dec 27 14:39:35.155730581 (527h10m35.026813626s) [noise=true , address=145.79.227.18:4007 , piece_id=YQ2LV2CREG7CRUT5X7RVUKQK2NEXND5DNKV7CCHZ5N2Y56BQQUEA, seg_pos={0 0}, upload=462392]: Uploading piece... Dec 27 14:39:35.156159168 (527h10m35.027242213s) [noise=true , address=145.79.227.10:4403 , piece_id=HIOVPOMOHA5INVIV6JTTCI7ZQKVUS25726JICLA5WDKI32OTP3PQ, seg_pos={0 0}, upload=462392]: Uploading piece... Dec 27 14:39:35.157452290 (527h10m35.028535325s) [noise=true , address=77.90.31.14:4662 , piece_id=FQ435MPZ6X7G5RQWUXTZTI7LSBYF2KTPWLM4K6AODCKUEA6E3UUA, seg_pos={0 0}, upload=462392]: Uploading piece... Dec 27 14:39:35.160601746 (527h10m35.031684792s) [noise=true , address=77.90.31.18:4805 , piece_id=PWPNOL5UJVHLV5Q64SKB6LKX24SUS5WK5CNAO4XZ7O3FPZ2PF7WQ, seg_pos={0 0}, upload=462392]: Uploading piece... Dec 27 14:39:35.166618984 (527h10m35.03770202s) [noise=true , address=45.91.116.13:4155 , piece_id=NW6FF4B52RKJFEYYWZH5ZQNLTTFSG2H4ZWNAPSCIPXQM6LWQZZAQ, seg_pos={0 0}, upload=462392]: Uploading piece... Dec 27 14:39:35.168085056 (527h10m35.039168091s) [noise=true , address=45.91.116.12:4228 , piece_id=QN5GY62LASOBMCTSHOSTMY6LDZBNVQ7KYA3EJR32IJ7BHW7K7D6Q, seg_pos={0 0}, upload=462392]: Uploading piece... Dec 27 14:39:36.824277451 (527h10m36.695360486s) [upload=462392]: Next returned nil segment Dec 27 14:39:36.824297880 (527h10m36.695380915s) [upload=462392]: Waiting for error group managing segments... Dec 27 14:39:37.153695049 (527h10m37.024778084s) [noise=true , address=145.79.227.45:4231 , piece_id=LVR4D2J2SNKCTGIUKMMIFE45SKSKCTRFXFWQ5MABGLXILJUN6UIQ, seg_pos={0 0}, upload=462392]: Done uploading piece. err: <nil> Dec 27 14:39:37.154867020 (527h10m37.025950055s) [noise=true , address=145.79.227.10:4403 , piece_id=HIOVPOMOHA5INVIV6JTTCI7ZQKVUS25726JICLA5WDKI32OTP3PQ, seg_pos={0 0}, upload=462392]: Done uploading piece. err: <nil> Dec 27 14:39:37.162541534 (527h10m37.033624579s) [noise=true , address=145.79.227.46:4603 , piece_id=73NJC7PAUTDZST3I6HS44TXZHDE7JYOQICVQSRMQTGQMBQMPYGZQ, seg_pos={0 0}, upload=462392]: Done uploading piece. err: <nil> Dec 27 14:39:37.176631633 (527h10m37.047714668s) [noise=true , address=145.79.227.18:4007 , piece_id=YQ2LV2CREG7CRUT5X7RVUKQK2NEXND5DNKV7CCHZ5N2Y56BQQUEA, seg_pos={0 0}, upload=462392]: Done uploading piece. err: <nil> Dec 27 14:39:37.181033666 (527h10m37.052116711s) [noise=true , address=145.79.227.42:4123 , piece_id=6NOG42HTTE5MUIM3RD47DPA7LGDIQ3E4MGF5NR3LS2HQOU4MIEEA, seg_pos={0 0}, upload=462392]: Done uploading piece. err: <nil> Dec 27 14:39:37.182330786 (527h10m37.053413831s) [noise=true , address=145.79.227.2:4113 , piece_id=EFTQU5HHSHQS6HPHOSMIVVUIHDS62IOG65MLN6WJEN2SNSXQFLWA, seg_pos={0 0}, upload=462392]: Done uploading piece. err: <nil> Dec 27 14:39:37.184363627 (527h10m37.055446672s) [noise=true , address=145.79.227.26:4655 , piece_id=L7223FKI5SQTGYT7SYQNSGCWOXHBAK7HEBCABPTQWW7NSG2FHH6A, seg_pos={0 0}, upload=462392]: Done uploading piece. err: <nil> Dec 27 14:39:37.418242372 (527h10m37.289325407s) [noise=true , address=45.91.116.6:4976 , piece_id=J4TMBLY35XHJZB7GZYCLGLFRTHPSTIYK4QBHBQMBZ7KYY7Q5AVQQ, seg_pos={0 0}, upload=462392]: Done uploading piece. err: <nil> Dec 27 14:39:37.420809731 (527h10m37.291892776s) [noise=true , address=77.90.31.14:4662 , piece_id=FQ435MPZ6X7G5RQWUXTZTI7LSBYF2KTPWLM4K6AODCKUEA6E3UUA, seg_pos={0 0}, upload=462392]: Done uploading piece. err: <nil> Dec 27 14:39:37.494097924 (527h10m37.365180959s) [noise=true , address=45.91.116.7:4010 , piece_id=MDERFXEOP7WCC7NPECA6QNLZSVTF3ZBCVXMKIS7DDNRV3TZRCCCA, seg_pos={0 0}, upload=462392]: Done uploading piece. err: <nil> Dec 27 14:39:37.518418275 (527h10m37.38950131s) [noise=true , address=45.91.116.13:4155 , piece_id=NW6FF4B52RKJFEYYWZH5ZQNLTTFSG2H4ZWNAPSCIPXQM6LWQZZAQ, seg_pos={0 0}, upload=462392]: Done uploading piece. err: <nil> Dec 27 14:39:37.748008344 (527h10m37.619091389s) [noise=true , address=77.90.31.18:4805 , piece_id=PWPNOL5UJVHLV5Q64SKB6LKX24SUS5WK5CNAO4XZ7O3FPZ2PF7WQ, seg_pos={0 0}, upload=462392]: Done uploading piece. err: <nil> Dec 27 14:39:37.772925761 (527h10m37.644008796s) [noise=true , address=45.91.116.4:4940 , piece_id=H2U4QZL7BTK7VWRY4KYLHINSW325Q2N4GPWCTMH2KNOCRIVZ7VJQ, seg_pos={0 0}, upload=462392]: Done uploading piece. err: <nil> Dec 27 14:39:37.772946721 (527h10m37.644029756s) [seg_pos={0 0}, upload=462392]: Segment reached optimal threshold of 13 pieces. Dec 27 14:39:37.773117736 (527h10m37.644200771s) [noise=true , address=145.79.227.35:4481 , piece_id=HTNUI4NFDQJU7BVCGXPBRMMOXG3U55RHXHGZVIJOEWJ7GQSJ2CZA, seg_pos={0 0}, upload=462392]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 27 14:39:37.773123648 (527h10m37.644206693s) [noise=true , address=145.79.227.35:4481 , piece_id=HTNUI4NFDQJU7BVCGXPBRMMOXG3U55RHXHGZVIJOEWJ7GQSJ2CZA, seg_pos={0 0}, upload=462392]: Long tail cancellation due to optimal threshold Dec 27 14:39:37.773221463 (527h10m37.644304498s) [noise=true , address=77.90.31.10:4520 , piece_id=TU7IIIV4Q3MYLNEUHGPOVKFU5POHKS57HM5PLYXTVYV4H2544XRQ, seg_pos={0 0}, upload=462392]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 27 14:39:37.773227947 (527h10m37.644310982s) [noise=true , address=77.90.31.10:4520 , piece_id=TU7IIIV4Q3MYLNEUHGPOVKFU5POHKS57HM5PLYXTVYV4H2544XRQ, seg_pos={0 0}, upload=462392]: Long tail cancellation due to optimal threshold Dec 27 14:39:37.773440531 (527h10m37.644523576s) [noise=true , address=45.91.116.12:4228 , piece_id=QN5GY62LASOBMCTSHOSTMY6LDZBNVQ7KYA3EJR32IJ7BHW7K7D6Q, seg_pos={0 0}, upload=462392]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:52930->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:52930->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->45.91.116.12:4228: use of closed network connection Dec 27 14:39:37.773495616 (527h10m37.644578651s) [noise=true , address=45.91.116.12:4228 , piece_id=QN5GY62LASOBMCTSHOSTMY6LDZBNVQ7KYA3EJR32IJ7BHW7K7D6Q, seg_pos={0 0}, upload=462392]: Long tail cancellation due to optimal threshold Dec 27 14:39:37.773529039 (527h10m37.644612084s) [seg_pos={0 0}, upload=462392]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 14:39:37.773698342 (527h10m37.644781397s) [upload=462392]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:39:37.773705325 (527h10m37.64478837s) [upload=462392]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:39:38.067957832 (527h10m37.939040878s) [upload=462392]: Done uploading segments... Dec 27 14:39:38.067982019 (527h10m37.939065064s) [upload=462392]: Upload finished. err: <nil> Duration: 6.017372666s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....XX......................................................] duration: 6.893716961s trace: https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a FAIL (2025-12-27 14:41:34 UTC) orbiter warning: context deadline exceeded Duration: 6.893741298s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [XX.X..XXXX.XXXXXXX.XXXXXX.XX.XXXXXX.XXXX.XXXXXXXXXXXX..XXXXX] duration: 5.001122556s trace: https://jaeger.dev.storj.io/trace/6b43d110632da573 FAIL (2025-12-27 14:41:28 UTC) orbiter warning: context deadline exceeded Duration: 5.001144468s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 8.606349166s trace: https://jaeger.dev.storj.io/trace/2f00e104e333c43f FAIL (2025-12-27 14:41:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== }, upload=462172]: Uploading piece... Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments... Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces. Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments... Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil> Duration: 8.606365778s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 33.50771668s trace: https://jaeger.dev.storj.io/trace/2f00e104e333c43f FAIL (2025-12-27 14:41:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== }, upload=462172]: Uploading piece... Dec 27 14:23:52.228052084 (526h54m52.099135129s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228569870 (526h54m52.099652915s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.228982526 (526h54m52.100065571s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229484542 (526h54m52.100567577s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.229989944 (526h54m52.101072989s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230467403 (526h54m52.101550438s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.230937939 (526h54m52.102020974s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.231768371 (526h54m52.102851416s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232385205 (526h54m52.10346825s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:52.232937757 (526h54m52.104020802s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Uploading piece... Dec 27 14:23:53.064921527 (526h54m52.936004562s) [upload=462172]: Next returned nil segment Dec 27 14:23:53.064937247 (526h54m52.936020282s) [upload=462172]: Waiting for error group managing segments... Dec 27 14:23:53.451979943 (526h54m53.323062989s) [noise=true , address=185.167.204.206:4067 , piece_id=ZNXGYY4ZMAHN5OWDRNA7SICFINWV2VES42WZB4WXSMFNL5M24FSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452245298 (526h54m53.323328333s) [noise=true , address=185.167.204.205:4031 , piece_id=MSCTNMDA24KZ7RJZCN3622XWQJ57UF7T4VXG5DFXFNAIU77REVQA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.452834570 (526h54m53.323917615s) [noise=true , address=185.167.204.205:4014 , piece_id=OOUDXYALWJAYTDXM6ZZ2TL4IATLPIUXEGJLG7XB2JWS6YXV375IA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453133290 (526h54m53.324216335s) [noise=true , address=185.167.204.206:4070 , piece_id=TYLPRDUE26XT46JKLMIGNMHWCRVSBPXKWA6Y4MA2SR6TAILO44KQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.453580251 (526h54m53.324663316s) [noise=true , address=185.167.204.206:4072 , piece_id=EJORI33Q44357WAKFSKNTECBVRCXYVUDGQV74LGO2RXWXDJZJ54A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454173050 (526h54m53.325256095s) [noise=true , address=185.167.204.206:4066 , piece_id=ZS3JP3J57Z7XI4AZQI4JGNVKKH3AF5NKTB72OF3R65J6NBZABOAQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454585044 (526h54m53.325668079s) [noise=true , address=185.167.204.205:4010 , piece_id=FLHQWPHM72SGS2HX3R2TIL5HJW5HO72WPK2NQDMOSQIDL2VYCFUQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.454954818 (526h54m53.326037863s) [noise=true , address=185.167.204.205:4020 , piece_id=FUBDSGOURX2V3WKCFNEINLTUN6JH6V33NFJOY6AWV3LU2OFYT62Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455576912 (526h54m53.326659957s) [noise=true , address=185.167.204.206:4047 , piece_id=7RVDADOWH5PBK2KOO7QWAQT2MUVDAALYZ65DPYJBZ3YDN6VVNUEQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.455804657 (526h54m53.326887712s) [noise=true , address=185.167.204.206:4043 , piece_id=DEHUFRWH5OL37ZW25D5R66ZEE7C4BI4EPIEYO6HURRZWOUIJUJWQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.456216099 (526h54m53.327299144s) [noise=true , address=185.167.204.206:4039 , piece_id=GK4JU6CZAQHWKYDSB2WEQVO6FCBLJWGQ4EPG24QLQZA34I4OQCFQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463315057 (526h54m53.334398453s) [noise=true , address=185.167.204.205:4034 , piece_id=3XUJGNFKLWLTQW7MSA2CNXEXPAP3MKBUDIFTAWOCX3ORLVYLEFSA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.463649905 (526h54m53.33473295s) [noise=true , address=185.167.204.205:4017 , piece_id=4GLJTSB6KWQLLBZHE7LUNP2QN6O3LAKC3WXHCYZCYEO6UPKO3TLQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.464203599 (526h54m53.335286644s) [noise=true , address=185.167.204.206:4057 , piece_id=VBOKUN5DYC7KLBXV5LLZCWWGFZGWS6LV67DEE3NLSNBDYVQ5X2ZQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467339641 (526h54m53.338422696s) [noise=true , address=185.167.204.206:4049 , piece_id=CO7M4DBPEHH3DAKPC2R7G7MGUA5HHT4JUVV743O4EZANWQQVDE2A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.467831888 (526h54m53.338914933s) [noise=true , address=185.167.204.206:4046 , piece_id=KQHAFGICYADH2LDTL36OK7DEE4NOJYNS2ZHCESRLBXGGUMB7AUUA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:53.469101085 (526h54m53.34018413s) [noise=true , address=185.167.204.205:4011 , piece_id=AWQ4ODXHCSC4KREXNW2UENAQOUPJENLGIJGERGESQHNEYCW6KYXQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.116176598 (526h54m53.987259633s) [noise=true , address=81.171.14.70:4031 , piece_id=T3MLDJ2I2JZSKE3QMFY3V5G32FU7K6ZCRGP257GACCN74JTO2SCQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.129478447 (526h54m54.000561482s) [noise=true , address=81.171.14.70:4027 , piece_id=KGWWRJVHIY7IETJ6LHG4SVMD26ZRE77B3HEN7674LH56DFNNV26Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.176888765 (526h54m54.0479718s) [noise=true , address=81.171.14.70:4036 , piece_id=FUWKBBICVV3FTHPZDYACJ4GYAQOEITGEFUJMALZDJFGYESESGB3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.226439356 (526h54m54.097522391s) [noise=true , address=81.171.14.70:4043 , piece_id=QL6FNZEVLDPHD7DT5CEMGB52MJMNUAI7RPSTSN42GFCMMNBPAGFA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.228387796 (526h54m54.099470841s) [noise=true , address=81.171.14.70:4046 , piece_id=ANQFXLWTWQIDLYH6LXPA7DCKJSEJL5DQ6JXUKKRUFNMXNRZBNTRA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.520867709 (526h54m54.391950744s) [noise=true , address=81.171.14.70:4026 , piece_id=HIOQ6Z2XKXPTW4RZQYLTJPAK57C4LVIYKJMFYE3LW35QIY3SF43Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.708483953 (526h54m54.579566998s) [noise=true , address=89.149.201.147:4002 , piece_id=DVGR3XEDA5TN6W4XQZVNUE7PAKZVE2PJITXZWRKBQC2PDO7GMGOA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:54.812477935 (526h54m54.68356098s) [noise=true , address=89.149.201.147:4007 , piece_id=GLY4EHKN37YNBKPHOTG4YYX3NFUM6TDMWKYMWCWYYRFQSZBPVEGA, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327599233 (526h54m55.198682278s) [noise=true , address=89.149.201.147:4004 , piece_id=IZCVMCIL4ALYIFHUACW32IKESXCN3FEHEODHXVLRG3JABITWS72A, seg_pos={0 0}, upload=462172]: Done uploading piece. err: <nil> Dec 27 14:23:55.327620925 (526h54m55.19870397s) [seg_pos={0 0}, upload=462172]: Segment reached optimal threshold of 26 pieces. Dec 27 14:23:55.327838108 (526h54m55.198921143s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121wRVh3zqETdzR1KQbRUBmnd11SsxFkFHngWG8uxyndwsbUXJz): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.327848358 (526h54m55.198931403s) [noise=true , address=89.149.201.147:4021 , piece_id=J5QKIK3ZJOYW45Q23XRPEI4AWNYYJLFDDEBDWAMXXJCMYYCLVYKQ, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.327967404 (526h54m55.199050449s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BnjusiXEfHYwTDL8soMkm8GYnhbkwy2oCAxVA7oiDyhFXozZ8): protocol: expected piece hash; context canceled; EOF Dec 27 14:23:55.328013332 (526h54m55.199096367s) [noise=true , address=81.171.14.70:4033 , piece_id=KWN2OZOIUYT2EG5GPPNQCA4TRQRKWV7QMEGNHAKQSCHLOY4XXV3Q, seg_pos={0 0}, upload=462172]: Long tail cancellation due to optimal threshold Dec 27 14:23:55.328033981 (526h54m55.199117026s) [seg_pos={0 0}, upload=462172]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 27 14:23:55.328118091 (526h54m55.199201126s) [upload=462172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:23:55.328123462 (526h54m55.199206497s) [upload=462172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:23:55.564051940 (526h54m55.435134975s) [upload=462172]: Done uploading segments... Dec 27 14:23:55.564101544 (526h54m55.435184589s) [upload=462172]: Upload finished. err: <nil> Duration: 33.507738031s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................X.............X..X...........X] duration: 1m0.002894346s trace: https://jaeger.dev.storj.io/trace/4d753dda46642d0d FAIL (2025-12-27 14:40:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd, Piece ID: 5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA) context deadline exceeded; piecestore: (Node ID: 12bbyRMajztw9frbd1V1meid4vQU1AC343NtZrg6qe27HUyw6bQ, Piece ID: ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ) context deadline exceeded; piecestore: (Node ID: 12cth5bdi5iAWhHcLxqN9c7GLdheS7SLQ3x5fEHG1MdgdXWFdD6, Piece ID: OCZIJHDS4AESHNCHUJIBVN47X336BIKXBLIQ4BVYSF7LAPWIS3KA) context deadline exceeded ======LOGS====== 660 (527h10m58.618722705s) [noise=true , address=77.90.31.19:4860 , piece_id=5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748001970 (527h10m58.619085005s) [noise=true , address=145.79.227.44:4205 , piece_id=HVNDI6C4SQZYPZIEEP7TC5RW6YNW5WGFDOJKQIIAN3UH3VGDXNIA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748340274 (527h10m58.619423319s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748682386 (527h10m58.619765421s) [noise=true , address=77.90.31.10:4530 , piece_id=K22R6V3CWZWVFKROB23NKYFC5R2Z5LT4KSXMTXFQUUROJKCJWESA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.748994651 (527h10m58.620077696s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.749319318 (527h10m58.620402363s) [noise=true , address=145.79.227.40:4060 , piece_id=OADO7RXPLZ4NLGUC2CCXNJQ6OEMU5JWTXCRY7WFWYKKGLMVU6YBA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.749662011 (527h10m58.620745046s) [noise=true , address=77.90.31.12:4601 , piece_id=LCBL5PH44YSNTX6W3UBMGJAGSOTBB6CKMBBD7E3GKDFQBUXMR3AA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.750250261 (527h10m58.621333296s) [noise=true , address=145.79.227.46:4633 , piece_id=U6GO7WKUMICS7QYSJZXZ6ZB33S2MYCV5JGEPVHEG7BHX5HGZL3XA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.750568958 (527h10m58.621651993s) [noise=true , address=145.79.227.23:4579 , piece_id=A3LOK5DMHEU3G3C6XYC2CWLOL7U4XJFXK4RV4A3PH4AXRHTA4EGQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.751689872 (527h10m58.622772917s) [noise=true , address=45.91.116.8:4049 , piece_id=ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.751758763 (527h10m58.622841809s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.753063619 (527h10m58.624146654s) [noise=true , address=145.79.227.36:4537 , piece_id=TMCDI7JYHU6MAPAGTLJTB2GSXMSBPJZJSPZWGAFF5I4VGEGNOMUQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.753568099 (527h10m58.624651134s) [noise=true , address=145.79.227.8:4337 , piece_id=BC5P5Y75VGZDTNDWEOC27UTK3TMWFRT7BNKQTJAYYHPYLYRXV2VQ, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.754895386 (527h10m58.625978421s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.755999769 (527h10m58.627082804s) [noise=true , address=77.90.31.21:4893 , piece_id=WYF4STSQ27E7KEMRIXKC4ILRHKN4UCNGH5NJFZ5SHOBKW3FNUCHA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:39:58.757486991 (527h10m58.628570037s) [noise=true , address=64.49.13.154:4933 , piece_id=V7STA7QTESDS5FDALZIPURQRCO6X4MONA2PTU5RIJHM34P5BQTNA, seg_pos={0 0}, upload=462397]: Uploading piece... Dec 27 14:40:00.120776880 (527h10m59.991859915s) [upload=462397]: Next returned nil segment Dec 27 14:40:00.120795044 (527h10m59.991878079s) [upload=462397]: Waiting for error group managing segments... Dec 27 14:40:00.475194542 (527h11m0.346277587s) [noise=true , address=64.49.13.154:4933 , piece_id=V7STA7QTESDS5FDALZIPURQRCO6X4MONA2PTU5RIJHM34P5BQTNA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.626897129 (527h11m0.497980174s) [noise=true , address=145.79.227.46:4633 , piece_id=U6GO7WKUMICS7QYSJZXZ6ZB33S2MYCV5JGEPVHEG7BHX5HGZL3XA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.634206858 (527h11m0.505289903s) [noise=true , address=145.79.227.34:4927 , piece_id=3PJB2HW34B5N5LADQYZ3FPNEIWBGKKGJEGADMPJIMWRG6HGQG36Q, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.636936777 (527h11m0.508019822s) [noise=true , address=145.79.227.36:4537 , piece_id=TMCDI7JYHU6MAPAGTLJTB2GSXMSBPJZJSPZWGAFF5I4VGEGNOMUQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.650341221 (527h11m0.521424266s) [noise=true , address=145.79.227.8:4337 , piece_id=BC5P5Y75VGZDTNDWEOC27UTK3TMWFRT7BNKQTJAYYHPYLYRXV2VQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.662509752 (527h11m0.533592797s) [noise=true , address=145.79.227.23:4579 , piece_id=A3LOK5DMHEU3G3C6XYC2CWLOL7U4XJFXK4RV4A3PH4AXRHTA4EGQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.820994414 (527h11m0.692077449s) [noise=true , address=77.90.31.12:4601 , piece_id=LCBL5PH44YSNTX6W3UBMGJAGSOTBB6CKMBBD7E3GKDFQBUXMR3AA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.908728394 (527h11m0.779811429s) [noise=true , address=45.91.116.8:4049 , piece_id=ANFY3JDYDLK4MMXU2SYZ4AV7UYXBRT4UR7QX5OTIUL4F54BFKOIQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:00.921937597 (527h11m0.793020642s) [noise=true , address=77.90.31.10:4530 , piece_id=K22R6V3CWZWVFKROB23NKYFC5R2Z5LT4KSXMTXFQUUROJKCJWESA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.180318328 (527h11m1.051401374s) [noise=true , address=77.90.31.14:4683 , piece_id=OCZIJHDS4AESHNCHUJIBVN47X336BIKXBLIQ4BVYSF7LAPWIS3KA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.467385073 (527h11m1.338468118s) [noise=true , address=77.90.31.19:4860 , piece_id=5XEJO56V7RJYIAS7FIS72BILS5EYMYUQ4NIQFVUZHQV333YI7XMA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.506047844 (527h11m1.377130919s) [noise=true , address=145.79.227.44:4205 , piece_id=HVNDI6C4SQZYPZIEEP7TC5RW6YNW5WGFDOJKQIIAN3UH3VGDXNIA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.551036449 (527h11m1.422119494s) [noise=true , address=145.79.227.40:4060 , piece_id=OADO7RXPLZ4NLGUC2CCXNJQ6OEMU5JWTXCRY7WFWYKKGLMVU6YBA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.567791785 (527h11m1.43887482s) [noise=true , address=77.90.31.21:4893 , piece_id=WYF4STSQ27E7KEMRIXKC4ILRHKN4UCNGH5NJFZ5SHOBKW3FNUCHA, seg_pos={0 0}, upload=462397]: Done uploading piece. err: <nil> Dec 27 14:40:01.567835187 (527h11m1.438918232s) [seg_pos={0 0}, upload=462397]: Segment reached optimal threshold of 14 pieces. Dec 27 14:40:01.568097526 (527h11m1.439180581s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568109259 (527h11m1.439192304s) [noise=true , address=145.79.227.30:4774 , piece_id=ZTALI4GJV2GFD2Z3M4FAZ34PPMI3RL5VBQHNZGDIF6Y24IBFY24A, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568506055 (527h11m1.4395891s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sfZTiYg2UFBmB45CB8FrBeMvmDoKMQ6Mt6zQHKgD5D8cwYe6q): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568516865 (527h11m1.43959991s) [noise=true , address=77.90.31.18:4834 , piece_id=RYYCLAYKQF4RA2VBY3YQFNQXOJTPRXGX4MMTGXOIE4B2TFDBTM6Q, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568673093 (527h11m1.439756148s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37270->64.49.13.169:4708: use of closed network connection Dec 27 14:40:01.568680948 (527h11m1.439763993s) [noise=true , address=64.49.13.169:4708 , piece_id=5IAPD2LEFLNTG4RHRW5IOZDJKJYV4MUMOIMU5RKAH2OGVHJHIXLQ, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568844680 (527h11m1.439927725s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FtB4bKTHntdUQxF8G5o9nT73oS2xB7fgD44BkRfKDNFyZnKCk): protocol: expected piece hash; context canceled; EOF Dec 27 14:40:01.568852474 (527h11m1.439935519s) [noise=true , address=77.90.31.17:4782 , piece_id=TCXFJX24GAGYQ4WRUBOHOUQKHUJV5SWGDLEHODH3O32LD64OWWYQ, seg_pos={0 0}, upload=462397]: Long tail cancellation due to optimal threshold Dec 27 14:40:01.568872633 (527h11m1.439955678s) [seg_pos={0 0}, upload=462397]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 14:40:01.568960319 (527h11m1.440043365s) [upload=462397]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 14:40:01.568966050 (527h11m1.440049095s) [upload=462397]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 14:40:01.862585703 (527h11m1.733668748s) [upload=462397]: Done uploading segments... Dec 27 14:40:01.862613346 (527h11m1.733696401s) [upload=462397]: Upload finished. err: <nil> Duration: 1m0.002904836s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)