Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................X...................................X] duration: 5.001143434s trace: https://jaeger.dev.storj.io/trace/bb932b4ff37ef93 FAIL (2025-12-27 05:08:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001171719s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X......................X.....................] duration: 53.508822224s trace: https://jaeger.dev.storj.io/trace/60d70ba2fe18218 FAIL (2025-12-27 05:07:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 4 (517h19m18.685148379s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814531152 (517h19m18.685614187s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814852964 (517h19m18.685936009s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815170900 (517h19m18.686253945s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815486661 (517h19m18.686569706s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815835786 (517h19m18.686918831s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.816207564 (517h19m18.687290609s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818225116 (517h19m18.689308151s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818325547 (517h19m18.689408592s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.821132431 (517h19m18.692215466s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.822845684 (517h19m18.693928719s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.824299251 (517h19m18.695382296s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.825754102 (517h19m18.696837147s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.827122548 (517h19m18.698205593s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.828453653 (517h19m18.699536688s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:20.114191076 (517h19m19.985274121s) [upload=453889]: Next returned nil segment Dec 27 04:48:20.114206816 (517h19m19.985289851s) [upload=453889]: Waiting for error group managing segments... Dec 27 04:48:20.444209056 (517h19m20.315292102s) [noise=true , address=64.49.13.154:4929 , piece_id=2EXG7BEQ6NKAG3TNAH3TZ3K5ZI4KNOQYF3OEZMFWVX6BBMWNVXUQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.474597952 (517h19m20.345680987s) [noise=true , address=64.49.13.27:4964 , piece_id=XAROGTY2JLVFUVDYT7N7ZCABVYDMXA45NZF36NBAV4H72M3PJSCA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.478601025 (517h19m20.34968406s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.888413067 (517h19m20.759496102s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.930510167 (517h19m20.801593212s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.007760305 (517h19m20.87884334s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.041631362 (517h19m20.912714407s) [noise=true , address=77.90.31.14:4691 , piece_id=HIFLW7CWEYK6A6BZAYPDZLIUNA26IFXSFNVYV2UK4VPKEWYUMYJA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.052961556 (517h19m20.924044601s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:23.606703638 (517h19m23.477786673s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.046520345 (517h19m25.91760338s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.518771011 (517h19m26.389854046s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.589256778 (517h19m26.460339823s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:27.044280620 (517h19m26.915363655s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789125901 (517h19m36.660208936s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789145728 (517h19m36.660228764s) [seg_pos={0 0}, upload=453889]: Segment reached optimal threshold of 14 pieces. Dec 27 04:48:36.789615964 (517h19m36.660699009s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: read tcp 188.166.225.170:50246->145.79.227.40:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789624180 (517h19m36.660707215s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789761231 (517h19m36.660844276s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: read tcp 188.166.225.170:45976->145.79.227.44:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789766831 (517h19m36.660849866s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789924742 (517h19m36.661007788s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51): protocol: read tcp 188.166.225.170:43134->145.79.227.43:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789933699 (517h19m36.661016734s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790128380 (517h19m36.661211425s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1): protocol: read tcp 188.166.225.170:46462->145.79.227.25:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.790137838 (517h19m36.661220873s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790156633 (517h19m36.661239679s) [seg_pos={0 0}, upload=453889]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:48:36.790241626 (517h19m36.661324661s) [upload=453889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:48:36.790246826 (517h19m36.661329861s) [upload=453889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:48:37.084198870 (517h19m36.955281915s) [upload=453889]: Done uploading segments... Dec 27 04:48:37.084219929 (517h19m36.955302974s) [upload=453889]: Upload finished. err: <nil> Duration: 53.508837302s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.587798084s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 05:07:02 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.587829292s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X......................X....................] duration: 54.736831078s trace: https://jaeger.dev.storj.io/trace/60d70ba2fe18218 FAIL (2025-12-27 05:07:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 4 (517h19m18.685148379s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814531152 (517h19m18.685614187s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814852964 (517h19m18.685936009s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815170900 (517h19m18.686253945s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815486661 (517h19m18.686569706s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815835786 (517h19m18.686918831s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.816207564 (517h19m18.687290609s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818225116 (517h19m18.689308151s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818325547 (517h19m18.689408592s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.821132431 (517h19m18.692215466s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.822845684 (517h19m18.693928719s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.824299251 (517h19m18.695382296s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.825754102 (517h19m18.696837147s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.827122548 (517h19m18.698205593s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.828453653 (517h19m18.699536688s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:20.114191076 (517h19m19.985274121s) [upload=453889]: Next returned nil segment Dec 27 04:48:20.114206816 (517h19m19.985289851s) [upload=453889]: Waiting for error group managing segments... Dec 27 04:48:20.444209056 (517h19m20.315292102s) [noise=true , address=64.49.13.154:4929 , piece_id=2EXG7BEQ6NKAG3TNAH3TZ3K5ZI4KNOQYF3OEZMFWVX6BBMWNVXUQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.474597952 (517h19m20.345680987s) [noise=true , address=64.49.13.27:4964 , piece_id=XAROGTY2JLVFUVDYT7N7ZCABVYDMXA45NZF36NBAV4H72M3PJSCA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.478601025 (517h19m20.34968406s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.888413067 (517h19m20.759496102s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.930510167 (517h19m20.801593212s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.007760305 (517h19m20.87884334s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.041631362 (517h19m20.912714407s) [noise=true , address=77.90.31.14:4691 , piece_id=HIFLW7CWEYK6A6BZAYPDZLIUNA26IFXSFNVYV2UK4VPKEWYUMYJA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.052961556 (517h19m20.924044601s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:23.606703638 (517h19m23.477786673s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.046520345 (517h19m25.91760338s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.518771011 (517h19m26.389854046s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.589256778 (517h19m26.460339823s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:27.044280620 (517h19m26.915363655s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789125901 (517h19m36.660208936s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789145728 (517h19m36.660228764s) [seg_pos={0 0}, upload=453889]: Segment reached optimal threshold of 14 pieces. Dec 27 04:48:36.789615964 (517h19m36.660699009s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: read tcp 188.166.225.170:50246->145.79.227.40:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789624180 (517h19m36.660707215s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789761231 (517h19m36.660844276s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: read tcp 188.166.225.170:45976->145.79.227.44:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789766831 (517h19m36.660849866s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789924742 (517h19m36.661007788s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51): protocol: read tcp 188.166.225.170:43134->145.79.227.43:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789933699 (517h19m36.661016734s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790128380 (517h19m36.661211425s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1): protocol: read tcp 188.166.225.170:46462->145.79.227.25:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.790137838 (517h19m36.661220873s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790156633 (517h19m36.661239679s) [seg_pos={0 0}, upload=453889]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:48:36.790241626 (517h19m36.661324661s) [upload=453889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:48:36.790246826 (517h19m36.661329861s) [upload=453889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:48:37.084198870 (517h19m36.955281915s) [upload=453889]: Done uploading segments... Dec 27 04:48:37.084219929 (517h19m36.955302974s) [upload=453889]: Upload finished. err: <nil> Duration: 54.736861556s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [....X...XX.XXX..XXX..XXXX...XX...X.XX....XXXX..X.X.XX..XXX..] duration: 5.97947102s trace: https://jaeger.dev.storj.io/trace/58492e92d78d8110 FAIL (2025-12-27 05:06:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: DQA2OFFXZW5H2WRUOKPJTNIPROIPUOJDKLFOUWVCNL4WWXD3IBZQ) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VMQ4G6FVMNOEBAVVK72376YBJP4LS3D3H6AWPYGBM6VVVONLGC7A) context deadline exceeded ======LOGS====== Dec 27 05:05:13.367310364 (517h36m13.238393399s) [upload=454126]: Starting upload Dec 27 05:05:13.367376039 (517h36m13.238459085s) [upload=454126]: Done starting upload Dec 27 05:05:13.367446844 (517h36m13.238529879s) [upload=454126]: Uploading segments... Dec 27 05:05:13.367572424 (517h36m13.238655469s) [upload=454126]: Got next segment. Inline: false Dec 27 05:05:13.367622819 (517h36m13.238705864s) [upload=454126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:05:13.367627057 (517h36m13.238710102s) [upload=454126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:05:13.613943524 (517h36m13.48502657s) [seg_pos={0 0}, upload=454126]: Begin upload segment... Dec 27 05:05:13.614004481 (517h36m13.485087526s) [seg_pos={0 0}, upload=454126]: Done begin upload segment. Dec 27 05:05:13.614032974 (517h36m13.485116029s) [noise=true , address=145.79.227.2:4114 , piece_id=3DHXRCJXKB7FEICX27KLM5YCRODSUAZ4NMHOKNF34RK5HF2ZGKGQ, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.614099421 (517h36m13.485182466s) [noise=true , address=145.79.227.47:4267 , piece_id=FIZSTVORVJT7YJLUUEZ5Q4Y2XM67QHELXISK55T7YERPMMRHHZGA, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.614215032 (517h36m13.485298077s) [noise=true , address=145.79.227.45:4229 , piece_id=OBUJAI2EDODLYTDMDP5QNEYRTVZ7DWTV3YTM3GXASPAJ74HEEXLQ, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.614776290 (517h36m13.485859335s) [noise=true , address=45.91.116.12:4226 , piece_id=KLZVWVXYCNRTIQHAJC62VG6N57STUII4TFE6I3YQLE6PJEBGPMQA, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.615192583 (517h36m13.486275628s) [noise=true , address=77.90.31.11:4554 , piece_id=ZHD3HL5VY2UGXTHJS6N7EI2S7W3ETJLTSH7VQTUPCUI227KMHHXA, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.615598295 (517h36m13.48668134s) [noise=true , address=145.79.227.38:4978 , piece_id=FVFUO5Q6WF7PJ2T7OTCLCXWZBCXKHMSKZ34VJAFELBPRDBIXYFLQ, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.616052750 (517h36m13.487135795s) [noise=true , address=45.91.116.7:4011 , piece_id=USBYP4QTAVQ4JD5V4RDHBVHLTKSL62BGKUPTCHYFYC3MZNOP4OWQ, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.616459956 (517h36m13.487543001s) [noise=true , address=145.79.227.11:4729 , piece_id=RUNUIV2QLAZC7ZCNK4OBYRCHSV5X37WIB64NK6HTYQFGRLFGNZPQ, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.616947484 (517h36m13.488030529s) [noise=true , address=45.91.116.13:4155 , piece_id=DATZ7JK3YUVGC4NDFJ3H7KPB6C3EMUWGLWGOFFRLQXAUIC6MTAKQ, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.617532388 (517h36m13.488615433s) [noise=true , address=145.79.227.3:4151 , piece_id=XYTSLGAP6XJFVGRUYSB4UQHQFU5O4TWFBRW7EB4CD3BJO5EUWUMQ, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.618110469 (517h36m13.489193504s) [noise=true , address=145.79.227.41:4085 , piece_id=EBXLCSENBLCFDTWWZB454DPLZT3V3IKISASOAVWRETKTDJQPHGBA, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.619840984 (517h36m13.490924029s) [noise=true , address=45.91.116.9:4263 , piece_id=DQA2OFFXZW5H2WRUOKPJTNIPROIPUOJDKLFOUWVCNL4WWXD3IBZQ, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.623035577 (517h36m13.494118622s) [noise=true , address=45.91.116.11:4117 , piece_id=VMQ4G6FVMNOEBAVVK72376YBJP4LS3D3H6AWPYGBM6VVVONLGC7A, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.627540635 (517h36m13.49862368s) [noise=true , address=145.79.227.15:4874 , piece_id=G27YG7HEFCXBOQUPFA2WOC6WCPQEE6WL5NRE4JG2VQ4WQF7FT7JA, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.630652151 (517h36m13.501735196s) [noise=true , address=145.79.227.35:4481 , piece_id=W3XB7GEAPUHQ6L23AIZNJTDERRARP4PANRHXDG2S7J7FOAHRICQA, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.632687245 (517h36m13.50377029s) [noise=true , address=145.79.227.39:4014 , piece_id=YEQC26WKBWAJONATBBWHHXW4AAKX3WEFKYRHVXJO7QYEBXMMCKPA, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:15.339347325 (517h36m15.21043037s) [upload=454126]: Next returned nil segment Dec 27 05:05:15.339365289 (517h36m15.210448324s) [upload=454126]: Waiting for error group managing segments... Dec 27 05:05:15.626704269 (517h36m15.497787304s) [noise=true , address=145.79.227.45:4229 , piece_id=OBUJAI2EDODLYTDMDP5QNEYRTVZ7DWTV3YTM3GXASPAJ74HEEXLQ, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.641299351 (517h36m15.512382396s) [noise=true , address=145.79.227.41:4085 , piece_id=EBXLCSENBLCFDTWWZB454DPLZT3V3IKISASOAVWRETKTDJQPHGBA, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.644820986 (517h36m15.515904021s) [noise=true , address=145.79.227.11:4729 , piece_id=RUNUIV2QLAZC7ZCNK4OBYRCHSV5X37WIB64NK6HTYQFGRLFGNZPQ, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.649586711 (517h36m15.520669756s) [noise=true , address=145.79.227.39:4014 , piece_id=YEQC26WKBWAJONATBBWHHXW4AAKX3WEFKYRHVXJO7QYEBXMMCKPA, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.652907154 (517h36m15.523990199s) [noise=true , address=145.79.227.15:4874 , piece_id=G27YG7HEFCXBOQUPFA2WOC6WCPQEE6WL5NRE4JG2VQ4WQF7FT7JA, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.661377954 (517h36m15.532460999s) [noise=true , address=145.79.227.47:4267 , piece_id=FIZSTVORVJT7YJLUUEZ5Q4Y2XM67QHELXISK55T7YERPMMRHHZGA, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.662854125 (517h36m15.53393717s) [noise=true , address=145.79.227.3:4151 , piece_id=XYTSLGAP6XJFVGRUYSB4UQHQFU5O4TWFBRW7EB4CD3BJO5EUWUMQ, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.667540088 (517h36m15.538623133s) [noise=true , address=145.79.227.2:4114 , piece_id=3DHXRCJXKB7FEICX27KLM5YCRODSUAZ4NMHOKNF34RK5HF2ZGKGQ, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.670723340 (517h36m15.541806385s) [noise=true , address=145.79.227.38:4978 , piece_id=FVFUO5Q6WF7PJ2T7OTCLCXWZBCXKHMSKZ34VJAFELBPRDBIXYFLQ, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.731634624 (517h36m15.60271767s) [noise=true , address=145.79.227.35:4481 , piece_id=W3XB7GEAPUHQ6L23AIZNJTDERRARP4PANRHXDG2S7J7FOAHRICQA, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.838695465 (517h36m15.70977851s) [noise=true , address=45.91.116.9:4263 , piece_id=DQA2OFFXZW5H2WRUOKPJTNIPROIPUOJDKLFOUWVCNL4WWXD3IBZQ, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.913179726 (517h36m15.784262761s) [noise=true , address=45.91.116.11:4117 , piece_id=VMQ4G6FVMNOEBAVVK72376YBJP4LS3D3H6AWPYGBM6VVVONLGC7A, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.926420218 (517h36m15.797503253s) [noise=true , address=77.90.31.11:4554 , piece_id=ZHD3HL5VY2UGXTHJS6N7EI2S7W3ETJLTSH7VQTUPCUI227KMHHXA, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.926438783 (517h36m15.797521818s) [seg_pos={0 0}, upload=454126]: Segment reached optimal threshold of 13 pieces. Dec 27 05:05:15.926712975 (517h36m15.79779601s) [noise=true , address=45.91.116.7:4011 , piece_id=USBYP4QTAVQ4JD5V4RDHBVHLTKSL62BGKUPTCHYFYC3MZNOP4OWQ, seg_pos={0 0}, upload=454126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 27 05:05:15.926719588 (517h36m15.797802623s) [noise=true , address=45.91.116.7:4011 , piece_id=USBYP4QTAVQ4JD5V4RDHBVHLTKSL62BGKUPTCHYFYC3MZNOP4OWQ, seg_pos={0 0}, upload=454126]: Long tail cancellation due to optimal threshold Dec 27 05:05:15.926933395 (517h36m15.79801643s) [noise=true , address=45.91.116.12:4226 , piece_id=KLZVWVXYCNRTIQHAJC62VG6N57STUII4TFE6I3YQLE6PJEBGPMQA, seg_pos={0 0}, upload=454126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 05:05:15.926942222 (517h36m15.798025257s) [noise=true , address=45.91.116.12:4226 , piece_id=KLZVWVXYCNRTIQHAJC62VG6N57STUII4TFE6I3YQLE6PJEBGPMQA, seg_pos={0 0}, upload=454126]: Long tail cancellation due to optimal threshold Dec 27 05:05:15.926955026 (517h36m15.798038061s) [noise=true , address=45.91.116.13:4155 , piece_id=DATZ7JK3YUVGC4NDFJ3H7KPB6C3EMUWGLWGOFFRLQXAUIC6MTAKQ, seg_pos={0 0}, upload=454126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 27 05:05:15.926961829 (517h36m15.798044864s) [noise=true , address=45.91.116.13:4155 , piece_id=DATZ7JK3YUVGC4NDFJ3H7KPB6C3EMUWGLWGOFFRLQXAUIC6MTAKQ, seg_pos={0 0}, upload=454126]: Long tail cancellation due to optimal threshold Dec 27 05:05:15.927003438 (517h36m15.798086473s) [seg_pos={0 0}, upload=454126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:05:15.927071167 (517h36m15.798154202s) [upload=454126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:05:15.927075717 (517h36m15.798158762s) [upload=454126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:05:16.231081934 (517h36m16.102164999s) [upload=454126]: Done uploading segments... Dec 27 05:05:16.231118373 (517h36m16.102201418s) [upload=454126]: Upload finished. err: <nil> Duration: 5.979500806s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [X....X...XX.XXX..XXX..XXXX...XX...X.XX....XXXX..X.X.XX..XXX.] duration: 5.741384462s trace: https://jaeger.dev.storj.io/trace/58492e92d78d8110 FAIL (2025-12-27 05:06:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: DQA2OFFXZW5H2WRUOKPJTNIPROIPUOJDKLFOUWVCNL4WWXD3IBZQ) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VMQ4G6FVMNOEBAVVK72376YBJP4LS3D3H6AWPYGBM6VVVONLGC7A) context deadline exceeded ======LOGS====== Dec 27 05:05:13.367310364 (517h36m13.238393399s) [upload=454126]: Starting upload Dec 27 05:05:13.367376039 (517h36m13.238459085s) [upload=454126]: Done starting upload Dec 27 05:05:13.367446844 (517h36m13.238529879s) [upload=454126]: Uploading segments... Dec 27 05:05:13.367572424 (517h36m13.238655469s) [upload=454126]: Got next segment. Inline: false Dec 27 05:05:13.367622819 (517h36m13.238705864s) [upload=454126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:05:13.367627057 (517h36m13.238710102s) [upload=454126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:05:13.613943524 (517h36m13.48502657s) [seg_pos={0 0}, upload=454126]: Begin upload segment... Dec 27 05:05:13.614004481 (517h36m13.485087526s) [seg_pos={0 0}, upload=454126]: Done begin upload segment. Dec 27 05:05:13.614032974 (517h36m13.485116029s) [noise=true , address=145.79.227.2:4114 , piece_id=3DHXRCJXKB7FEICX27KLM5YCRODSUAZ4NMHOKNF34RK5HF2ZGKGQ, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.614099421 (517h36m13.485182466s) [noise=true , address=145.79.227.47:4267 , piece_id=FIZSTVORVJT7YJLUUEZ5Q4Y2XM67QHELXISK55T7YERPMMRHHZGA, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.614215032 (517h36m13.485298077s) [noise=true , address=145.79.227.45:4229 , piece_id=OBUJAI2EDODLYTDMDP5QNEYRTVZ7DWTV3YTM3GXASPAJ74HEEXLQ, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.614776290 (517h36m13.485859335s) [noise=true , address=45.91.116.12:4226 , piece_id=KLZVWVXYCNRTIQHAJC62VG6N57STUII4TFE6I3YQLE6PJEBGPMQA, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.615192583 (517h36m13.486275628s) [noise=true , address=77.90.31.11:4554 , piece_id=ZHD3HL5VY2UGXTHJS6N7EI2S7W3ETJLTSH7VQTUPCUI227KMHHXA, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.615598295 (517h36m13.48668134s) [noise=true , address=145.79.227.38:4978 , piece_id=FVFUO5Q6WF7PJ2T7OTCLCXWZBCXKHMSKZ34VJAFELBPRDBIXYFLQ, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.616052750 (517h36m13.487135795s) [noise=true , address=45.91.116.7:4011 , piece_id=USBYP4QTAVQ4JD5V4RDHBVHLTKSL62BGKUPTCHYFYC3MZNOP4OWQ, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.616459956 (517h36m13.487543001s) [noise=true , address=145.79.227.11:4729 , piece_id=RUNUIV2QLAZC7ZCNK4OBYRCHSV5X37WIB64NK6HTYQFGRLFGNZPQ, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.616947484 (517h36m13.488030529s) [noise=true , address=45.91.116.13:4155 , piece_id=DATZ7JK3YUVGC4NDFJ3H7KPB6C3EMUWGLWGOFFRLQXAUIC6MTAKQ, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.617532388 (517h36m13.488615433s) [noise=true , address=145.79.227.3:4151 , piece_id=XYTSLGAP6XJFVGRUYSB4UQHQFU5O4TWFBRW7EB4CD3BJO5EUWUMQ, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.618110469 (517h36m13.489193504s) [noise=true , address=145.79.227.41:4085 , piece_id=EBXLCSENBLCFDTWWZB454DPLZT3V3IKISASOAVWRETKTDJQPHGBA, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.619840984 (517h36m13.490924029s) [noise=true , address=45.91.116.9:4263 , piece_id=DQA2OFFXZW5H2WRUOKPJTNIPROIPUOJDKLFOUWVCNL4WWXD3IBZQ, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.623035577 (517h36m13.494118622s) [noise=true , address=45.91.116.11:4117 , piece_id=VMQ4G6FVMNOEBAVVK72376YBJP4LS3D3H6AWPYGBM6VVVONLGC7A, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.627540635 (517h36m13.49862368s) [noise=true , address=145.79.227.15:4874 , piece_id=G27YG7HEFCXBOQUPFA2WOC6WCPQEE6WL5NRE4JG2VQ4WQF7FT7JA, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.630652151 (517h36m13.501735196s) [noise=true , address=145.79.227.35:4481 , piece_id=W3XB7GEAPUHQ6L23AIZNJTDERRARP4PANRHXDG2S7J7FOAHRICQA, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.632687245 (517h36m13.50377029s) [noise=true , address=145.79.227.39:4014 , piece_id=YEQC26WKBWAJONATBBWHHXW4AAKX3WEFKYRHVXJO7QYEBXMMCKPA, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:15.339347325 (517h36m15.21043037s) [upload=454126]: Next returned nil segment Dec 27 05:05:15.339365289 (517h36m15.210448324s) [upload=454126]: Waiting for error group managing segments... Dec 27 05:05:15.626704269 (517h36m15.497787304s) [noise=true , address=145.79.227.45:4229 , piece_id=OBUJAI2EDODLYTDMDP5QNEYRTVZ7DWTV3YTM3GXASPAJ74HEEXLQ, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.641299351 (517h36m15.512382396s) [noise=true , address=145.79.227.41:4085 , piece_id=EBXLCSENBLCFDTWWZB454DPLZT3V3IKISASOAVWRETKTDJQPHGBA, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.644820986 (517h36m15.515904021s) [noise=true , address=145.79.227.11:4729 , piece_id=RUNUIV2QLAZC7ZCNK4OBYRCHSV5X37WIB64NK6HTYQFGRLFGNZPQ, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.649586711 (517h36m15.520669756s) [noise=true , address=145.79.227.39:4014 , piece_id=YEQC26WKBWAJONATBBWHHXW4AAKX3WEFKYRHVXJO7QYEBXMMCKPA, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.652907154 (517h36m15.523990199s) [noise=true , address=145.79.227.15:4874 , piece_id=G27YG7HEFCXBOQUPFA2WOC6WCPQEE6WL5NRE4JG2VQ4WQF7FT7JA, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.661377954 (517h36m15.532460999s) [noise=true , address=145.79.227.47:4267 , piece_id=FIZSTVORVJT7YJLUUEZ5Q4Y2XM67QHELXISK55T7YERPMMRHHZGA, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.662854125 (517h36m15.53393717s) [noise=true , address=145.79.227.3:4151 , piece_id=XYTSLGAP6XJFVGRUYSB4UQHQFU5O4TWFBRW7EB4CD3BJO5EUWUMQ, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.667540088 (517h36m15.538623133s) [noise=true , address=145.79.227.2:4114 , piece_id=3DHXRCJXKB7FEICX27KLM5YCRODSUAZ4NMHOKNF34RK5HF2ZGKGQ, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.670723340 (517h36m15.541806385s) [noise=true , address=145.79.227.38:4978 , piece_id=FVFUO5Q6WF7PJ2T7OTCLCXWZBCXKHMSKZ34VJAFELBPRDBIXYFLQ, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.731634624 (517h36m15.60271767s) [noise=true , address=145.79.227.35:4481 , piece_id=W3XB7GEAPUHQ6L23AIZNJTDERRARP4PANRHXDG2S7J7FOAHRICQA, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.838695465 (517h36m15.70977851s) [noise=true , address=45.91.116.9:4263 , piece_id=DQA2OFFXZW5H2WRUOKPJTNIPROIPUOJDKLFOUWVCNL4WWXD3IBZQ, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.913179726 (517h36m15.784262761s) [noise=true , address=45.91.116.11:4117 , piece_id=VMQ4G6FVMNOEBAVVK72376YBJP4LS3D3H6AWPYGBM6VVVONLGC7A, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.926420218 (517h36m15.797503253s) [noise=true , address=77.90.31.11:4554 , piece_id=ZHD3HL5VY2UGXTHJS6N7EI2S7W3ETJLTSH7VQTUPCUI227KMHHXA, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.926438783 (517h36m15.797521818s) [seg_pos={0 0}, upload=454126]: Segment reached optimal threshold of 13 pieces. Dec 27 05:05:15.926712975 (517h36m15.79779601s) [noise=true , address=45.91.116.7:4011 , piece_id=USBYP4QTAVQ4JD5V4RDHBVHLTKSL62BGKUPTCHYFYC3MZNOP4OWQ, seg_pos={0 0}, upload=454126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 27 05:05:15.926719588 (517h36m15.797802623s) [noise=true , address=45.91.116.7:4011 , piece_id=USBYP4QTAVQ4JD5V4RDHBVHLTKSL62BGKUPTCHYFYC3MZNOP4OWQ, seg_pos={0 0}, upload=454126]: Long tail cancellation due to optimal threshold Dec 27 05:05:15.926933395 (517h36m15.79801643s) [noise=true , address=45.91.116.12:4226 , piece_id=KLZVWVXYCNRTIQHAJC62VG6N57STUII4TFE6I3YQLE6PJEBGPMQA, seg_pos={0 0}, upload=454126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 05:05:15.926942222 (517h36m15.798025257s) [noise=true , address=45.91.116.12:4226 , piece_id=KLZVWVXYCNRTIQHAJC62VG6N57STUII4TFE6I3YQLE6PJEBGPMQA, seg_pos={0 0}, upload=454126]: Long tail cancellation due to optimal threshold Dec 27 05:05:15.926955026 (517h36m15.798038061s) [noise=true , address=45.91.116.13:4155 , piece_id=DATZ7JK3YUVGC4NDFJ3H7KPB6C3EMUWGLWGOFFRLQXAUIC6MTAKQ, seg_pos={0 0}, upload=454126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 27 05:05:15.926961829 (517h36m15.798044864s) [noise=true , address=45.91.116.13:4155 , piece_id=DATZ7JK3YUVGC4NDFJ3H7KPB6C3EMUWGLWGOFFRLQXAUIC6MTAKQ, seg_pos={0 0}, upload=454126]: Long tail cancellation due to optimal threshold Dec 27 05:05:15.927003438 (517h36m15.798086473s) [seg_pos={0 0}, upload=454126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:05:15.927071167 (517h36m15.798154202s) [upload=454126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:05:15.927075717 (517h36m15.798158762s) [upload=454126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:05:16.231081934 (517h36m16.102164999s) [upload=454126]: Done uploading segments... Dec 27 05:05:16.231118373 (517h36m16.102201418s) [upload=454126]: Upload finished. err: <nil> Duration: 5.741406194s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 3.516867732s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 05:06:06 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.51695084s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X......................X...................] duration: 50.64236305s trace: https://jaeger.dev.storj.io/trace/60d70ba2fe18218 FAIL (2025-12-27 05:05:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 4 (517h19m18.685148379s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814531152 (517h19m18.685614187s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814852964 (517h19m18.685936009s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815170900 (517h19m18.686253945s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815486661 (517h19m18.686569706s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815835786 (517h19m18.686918831s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.816207564 (517h19m18.687290609s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818225116 (517h19m18.689308151s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818325547 (517h19m18.689408592s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.821132431 (517h19m18.692215466s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.822845684 (517h19m18.693928719s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.824299251 (517h19m18.695382296s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.825754102 (517h19m18.696837147s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.827122548 (517h19m18.698205593s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.828453653 (517h19m18.699536688s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:20.114191076 (517h19m19.985274121s) [upload=453889]: Next returned nil segment Dec 27 04:48:20.114206816 (517h19m19.985289851s) [upload=453889]: Waiting for error group managing segments... Dec 27 04:48:20.444209056 (517h19m20.315292102s) [noise=true , address=64.49.13.154:4929 , piece_id=2EXG7BEQ6NKAG3TNAH3TZ3K5ZI4KNOQYF3OEZMFWVX6BBMWNVXUQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.474597952 (517h19m20.345680987s) [noise=true , address=64.49.13.27:4964 , piece_id=XAROGTY2JLVFUVDYT7N7ZCABVYDMXA45NZF36NBAV4H72M3PJSCA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.478601025 (517h19m20.34968406s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.888413067 (517h19m20.759496102s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.930510167 (517h19m20.801593212s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.007760305 (517h19m20.87884334s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.041631362 (517h19m20.912714407s) [noise=true , address=77.90.31.14:4691 , piece_id=HIFLW7CWEYK6A6BZAYPDZLIUNA26IFXSFNVYV2UK4VPKEWYUMYJA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.052961556 (517h19m20.924044601s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:23.606703638 (517h19m23.477786673s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.046520345 (517h19m25.91760338s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.518771011 (517h19m26.389854046s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.589256778 (517h19m26.460339823s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:27.044280620 (517h19m26.915363655s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789125901 (517h19m36.660208936s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789145728 (517h19m36.660228764s) [seg_pos={0 0}, upload=453889]: Segment reached optimal threshold of 14 pieces. Dec 27 04:48:36.789615964 (517h19m36.660699009s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: read tcp 188.166.225.170:50246->145.79.227.40:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789624180 (517h19m36.660707215s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789761231 (517h19m36.660844276s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: read tcp 188.166.225.170:45976->145.79.227.44:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789766831 (517h19m36.660849866s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789924742 (517h19m36.661007788s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51): protocol: read tcp 188.166.225.170:43134->145.79.227.43:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789933699 (517h19m36.661016734s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790128380 (517h19m36.661211425s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1): protocol: read tcp 188.166.225.170:46462->145.79.227.25:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.790137838 (517h19m36.661220873s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790156633 (517h19m36.661239679s) [seg_pos={0 0}, upload=453889]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:48:36.790241626 (517h19m36.661324661s) [upload=453889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:48:36.790246826 (517h19m36.661329861s) [upload=453889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:48:37.084198870 (517h19m36.955281915s) [upload=453889]: Done uploading segments... Dec 27 04:48:37.084219929 (517h19m36.955302974s) [upload=453889]: Upload finished. err: <nil> Duration: 50.642382246s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.X....X...XX.XXX..XXX..XXXX...XX...X.XX....XXXX..X.X.XX..XXX] duration: 30.00401373s trace: https://jaeger.dev.storj.io/trace/58492e92d78d8110 FAIL (2025-12-27 05:05:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: DQA2OFFXZW5H2WRUOKPJTNIPROIPUOJDKLFOUWVCNL4WWXD3IBZQ) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VMQ4G6FVMNOEBAVVK72376YBJP4LS3D3H6AWPYGBM6VVVONLGC7A) context deadline exceeded ======LOGS====== Dec 27 05:05:13.367310364 (517h36m13.238393399s) [upload=454126]: Starting upload Dec 27 05:05:13.367376039 (517h36m13.238459085s) [upload=454126]: Done starting upload Dec 27 05:05:13.367446844 (517h36m13.238529879s) [upload=454126]: Uploading segments... Dec 27 05:05:13.367572424 (517h36m13.238655469s) [upload=454126]: Got next segment. Inline: false Dec 27 05:05:13.367622819 (517h36m13.238705864s) [upload=454126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:05:13.367627057 (517h36m13.238710102s) [upload=454126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:05:13.613943524 (517h36m13.48502657s) [seg_pos={0 0}, upload=454126]: Begin upload segment... Dec 27 05:05:13.614004481 (517h36m13.485087526s) [seg_pos={0 0}, upload=454126]: Done begin upload segment. Dec 27 05:05:13.614032974 (517h36m13.485116029s) [noise=true , address=145.79.227.2:4114 , piece_id=3DHXRCJXKB7FEICX27KLM5YCRODSUAZ4NMHOKNF34RK5HF2ZGKGQ, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.614099421 (517h36m13.485182466s) [noise=true , address=145.79.227.47:4267 , piece_id=FIZSTVORVJT7YJLUUEZ5Q4Y2XM67QHELXISK55T7YERPMMRHHZGA, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.614215032 (517h36m13.485298077s) [noise=true , address=145.79.227.45:4229 , piece_id=OBUJAI2EDODLYTDMDP5QNEYRTVZ7DWTV3YTM3GXASPAJ74HEEXLQ, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.614776290 (517h36m13.485859335s) [noise=true , address=45.91.116.12:4226 , piece_id=KLZVWVXYCNRTIQHAJC62VG6N57STUII4TFE6I3YQLE6PJEBGPMQA, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.615192583 (517h36m13.486275628s) [noise=true , address=77.90.31.11:4554 , piece_id=ZHD3HL5VY2UGXTHJS6N7EI2S7W3ETJLTSH7VQTUPCUI227KMHHXA, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.615598295 (517h36m13.48668134s) [noise=true , address=145.79.227.38:4978 , piece_id=FVFUO5Q6WF7PJ2T7OTCLCXWZBCXKHMSKZ34VJAFELBPRDBIXYFLQ, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.616052750 (517h36m13.487135795s) [noise=true , address=45.91.116.7:4011 , piece_id=USBYP4QTAVQ4JD5V4RDHBVHLTKSL62BGKUPTCHYFYC3MZNOP4OWQ, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.616459956 (517h36m13.487543001s) [noise=true , address=145.79.227.11:4729 , piece_id=RUNUIV2QLAZC7ZCNK4OBYRCHSV5X37WIB64NK6HTYQFGRLFGNZPQ, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.616947484 (517h36m13.488030529s) [noise=true , address=45.91.116.13:4155 , piece_id=DATZ7JK3YUVGC4NDFJ3H7KPB6C3EMUWGLWGOFFRLQXAUIC6MTAKQ, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.617532388 (517h36m13.488615433s) [noise=true , address=145.79.227.3:4151 , piece_id=XYTSLGAP6XJFVGRUYSB4UQHQFU5O4TWFBRW7EB4CD3BJO5EUWUMQ, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.618110469 (517h36m13.489193504s) [noise=true , address=145.79.227.41:4085 , piece_id=EBXLCSENBLCFDTWWZB454DPLZT3V3IKISASOAVWRETKTDJQPHGBA, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.619840984 (517h36m13.490924029s) [noise=true , address=45.91.116.9:4263 , piece_id=DQA2OFFXZW5H2WRUOKPJTNIPROIPUOJDKLFOUWVCNL4WWXD3IBZQ, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.623035577 (517h36m13.494118622s) [noise=true , address=45.91.116.11:4117 , piece_id=VMQ4G6FVMNOEBAVVK72376YBJP4LS3D3H6AWPYGBM6VVVONLGC7A, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.627540635 (517h36m13.49862368s) [noise=true , address=145.79.227.15:4874 , piece_id=G27YG7HEFCXBOQUPFA2WOC6WCPQEE6WL5NRE4JG2VQ4WQF7FT7JA, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.630652151 (517h36m13.501735196s) [noise=true , address=145.79.227.35:4481 , piece_id=W3XB7GEAPUHQ6L23AIZNJTDERRARP4PANRHXDG2S7J7FOAHRICQA, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:13.632687245 (517h36m13.50377029s) [noise=true , address=145.79.227.39:4014 , piece_id=YEQC26WKBWAJONATBBWHHXW4AAKX3WEFKYRHVXJO7QYEBXMMCKPA, seg_pos={0 0}, upload=454126]: Uploading piece... Dec 27 05:05:15.339347325 (517h36m15.21043037s) [upload=454126]: Next returned nil segment Dec 27 05:05:15.339365289 (517h36m15.210448324s) [upload=454126]: Waiting for error group managing segments... Dec 27 05:05:15.626704269 (517h36m15.497787304s) [noise=true , address=145.79.227.45:4229 , piece_id=OBUJAI2EDODLYTDMDP5QNEYRTVZ7DWTV3YTM3GXASPAJ74HEEXLQ, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.641299351 (517h36m15.512382396s) [noise=true , address=145.79.227.41:4085 , piece_id=EBXLCSENBLCFDTWWZB454DPLZT3V3IKISASOAVWRETKTDJQPHGBA, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.644820986 (517h36m15.515904021s) [noise=true , address=145.79.227.11:4729 , piece_id=RUNUIV2QLAZC7ZCNK4OBYRCHSV5X37WIB64NK6HTYQFGRLFGNZPQ, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.649586711 (517h36m15.520669756s) [noise=true , address=145.79.227.39:4014 , piece_id=YEQC26WKBWAJONATBBWHHXW4AAKX3WEFKYRHVXJO7QYEBXMMCKPA, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.652907154 (517h36m15.523990199s) [noise=true , address=145.79.227.15:4874 , piece_id=G27YG7HEFCXBOQUPFA2WOC6WCPQEE6WL5NRE4JG2VQ4WQF7FT7JA, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.661377954 (517h36m15.532460999s) [noise=true , address=145.79.227.47:4267 , piece_id=FIZSTVORVJT7YJLUUEZ5Q4Y2XM67QHELXISK55T7YERPMMRHHZGA, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.662854125 (517h36m15.53393717s) [noise=true , address=145.79.227.3:4151 , piece_id=XYTSLGAP6XJFVGRUYSB4UQHQFU5O4TWFBRW7EB4CD3BJO5EUWUMQ, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.667540088 (517h36m15.538623133s) [noise=true , address=145.79.227.2:4114 , piece_id=3DHXRCJXKB7FEICX27KLM5YCRODSUAZ4NMHOKNF34RK5HF2ZGKGQ, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.670723340 (517h36m15.541806385s) [noise=true , address=145.79.227.38:4978 , piece_id=FVFUO5Q6WF7PJ2T7OTCLCXWZBCXKHMSKZ34VJAFELBPRDBIXYFLQ, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.731634624 (517h36m15.60271767s) [noise=true , address=145.79.227.35:4481 , piece_id=W3XB7GEAPUHQ6L23AIZNJTDERRARP4PANRHXDG2S7J7FOAHRICQA, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.838695465 (517h36m15.70977851s) [noise=true , address=45.91.116.9:4263 , piece_id=DQA2OFFXZW5H2WRUOKPJTNIPROIPUOJDKLFOUWVCNL4WWXD3IBZQ, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.913179726 (517h36m15.784262761s) [noise=true , address=45.91.116.11:4117 , piece_id=VMQ4G6FVMNOEBAVVK72376YBJP4LS3D3H6AWPYGBM6VVVONLGC7A, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.926420218 (517h36m15.797503253s) [noise=true , address=77.90.31.11:4554 , piece_id=ZHD3HL5VY2UGXTHJS6N7EI2S7W3ETJLTSH7VQTUPCUI227KMHHXA, seg_pos={0 0}, upload=454126]: Done uploading piece. err: <nil> Dec 27 05:05:15.926438783 (517h36m15.797521818s) [seg_pos={0 0}, upload=454126]: Segment reached optimal threshold of 13 pieces. Dec 27 05:05:15.926712975 (517h36m15.79779601s) [noise=true , address=45.91.116.7:4011 , piece_id=USBYP4QTAVQ4JD5V4RDHBVHLTKSL62BGKUPTCHYFYC3MZNOP4OWQ, seg_pos={0 0}, upload=454126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 27 05:05:15.926719588 (517h36m15.797802623s) [noise=true , address=45.91.116.7:4011 , piece_id=USBYP4QTAVQ4JD5V4RDHBVHLTKSL62BGKUPTCHYFYC3MZNOP4OWQ, seg_pos={0 0}, upload=454126]: Long tail cancellation due to optimal threshold Dec 27 05:05:15.926933395 (517h36m15.79801643s) [noise=true , address=45.91.116.12:4226 , piece_id=KLZVWVXYCNRTIQHAJC62VG6N57STUII4TFE6I3YQLE6PJEBGPMQA, seg_pos={0 0}, upload=454126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 05:05:15.926942222 (517h36m15.798025257s) [noise=true , address=45.91.116.12:4226 , piece_id=KLZVWVXYCNRTIQHAJC62VG6N57STUII4TFE6I3YQLE6PJEBGPMQA, seg_pos={0 0}, upload=454126]: Long tail cancellation due to optimal threshold Dec 27 05:05:15.926955026 (517h36m15.798038061s) [noise=true , address=45.91.116.13:4155 , piece_id=DATZ7JK3YUVGC4NDFJ3H7KPB6C3EMUWGLWGOFFRLQXAUIC6MTAKQ, seg_pos={0 0}, upload=454126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 27 05:05:15.926961829 (517h36m15.798044864s) [noise=true , address=45.91.116.13:4155 , piece_id=DATZ7JK3YUVGC4NDFJ3H7KPB6C3EMUWGLWGOFFRLQXAUIC6MTAKQ, seg_pos={0 0}, upload=454126]: Long tail cancellation due to optimal threshold Dec 27 05:05:15.927003438 (517h36m15.798086473s) [seg_pos={0 0}, upload=454126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:05:15.927071167 (517h36m15.798154202s) [upload=454126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:05:15.927075717 (517h36m15.798158762s) [upload=454126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:05:16.231081934 (517h36m16.102164999s) [upload=454126]: Done uploading segments... Dec 27 05:05:16.231118373 (517h36m16.102201418s) [upload=454126]: Upload finished. err: <nil> Duration: 30.00402873s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [X.X....X...XX.XXX..XXX..XXXX...XX...X.XX....XXXX..X.X.XX..XX] duration: 30.00265022s trace: https://jaeger.dev.storj.io/trace/6aaa4fd626fbdc20 FAIL (2025-12-27 05:05:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: QBAZSTRJINY47TBQVOJ2QC3RIBOQ4ULA3EYU4VV2HJ5PYX7UWGGQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: NNJYU4XCWUPGB6VMEKUU3MOF7YHZG3RYZBTBNOX7QUPKCGED2CTQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: 5WKWGRREIPFZPTEOOA53XR4IPPD6HDGRIED3DG4VQSV62NKTUJRA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: HK3QXUPBOCCWU6CP246SVF3G56DCUPXL3SN7DHKAEKEI3T4D24QQ) context deadline exceeded ======LOGS====== Dec 27 05:04:43.090062545 (517h35m42.961145601s) [upload=454119]: Starting upload Dec 27 05:04:43.090150664 (517h35m42.961233699s) [upload=454119]: Done starting upload Dec 27 05:04:43.090206871 (517h35m42.961289916s) [upload=454119]: Uploading segments... Dec 27 05:04:43.090514366 (517h35m42.961597401s) [upload=454119]: Got next segment. Inline: false Dec 27 05:04:43.090560845 (517h35m42.96164388s) [upload=454119]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:04:43.090566365 (517h35m42.9616494s) [upload=454119]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:04:43.337140321 (517h35m43.208223367s) [seg_pos={0 0}, upload=454119]: Begin upload segment... Dec 27 05:04:43.337202350 (517h35m43.208285385s) [seg_pos={0 0}, upload=454119]: Done begin upload segment. Dec 27 05:04:43.337230644 (517h35m43.208313689s) [noise=true , address=145.79.227.45:4229 , piece_id=LHDM324WTKLUHKJXKR32LOVM4PBEDISYKR3EUVS67W6KO56CSMIQ, seg_pos={0 0}, upload=454119]: Uploading piece... Dec 27 05:04:43.337240322 (517h35m43.208323357s) [noise=true , address=45.91.116.4:4940 , piece_id=SEL73K4UYC6EYPU7KTWIYD5B5UBE3ZUBKVEJ4XEKHVAOTSAOANIQ, seg_pos={0 0}, upload=454119]: Uploading piece... Dec 27 05:04:43.338036017 (517h35m43.209119052s) [noise=true , address=145.79.227.23:4546 , piece_id=DT6EHZKNKMTRNCUSVBZPXGCJYLJNJAGKLGVJ62UZZUPDG7YFGAGQ, seg_pos={0 0}, upload=454119]: Uploading piece... Dec 27 05:04:43.338716153 (517h35m43.209799208s) [noise=true , address=77.90.31.16:4736 , piece_id=BHW4TX3SPKGYQWUYEBM53XTCXEKYGVXBKEX5J6GRQC7G5X5N4PBA, seg_pos={0 0}, upload=454119]: Uploading piece... Dec 27 05:04:43.338726642 (517h35m43.209809688s) [noise=true , address=145.79.227.28:4725 , piece_id=4LSPEZDR2XOHZVDRW33D7MIMEPCV5ADDGJJ3WCQ7YGJDWP437H4Q, seg_pos={0 0}, upload=454119]: Uploading piece... Dec 27 05:04:43.339074044 (517h35m43.210157089s) [noise=true , address=145.79.227.39:4014 , piece_id=SIFRERR4DYHHQJDJY7LROKSOS2EI7DMNTI63NLDG4VYO6HYEOPLA, seg_pos={0 0}, upload=454119]: Uploading piece... Dec 27 05:04:43.339367753 (517h35m43.210450798s) [noise=true , address=45.91.116.7:4009 , piece_id=QBAZSTRJINY47TBQVOJ2QC3RIBOQ4ULA3EYU4VV2HJ5PYX7UWGGQ, seg_pos={0 0}, upload=454119]: Uploading piece... Dec 27 05:04:43.339503361 (517h35m43.210586406s) [noise=true , address=145.79.227.33:4871 , piece_id=IVGVIUQI35L2JXVDJW46S5QSX2VRQJAFGQQBQ27IL6KJ2VREECJQ, seg_pos={0 0}, upload=454119]: Uploading piece... Dec 27 05:04:43.339934222 (517h35m43.211017277s) [noise=true , address=45.91.116.8:4045 , piece_id=NNJYU4XCWUPGB6VMEKUU3MOF7YHZG3RYZBTBNOX7QUPKCGED2CTQ, seg_pos={0 0}, upload=454119]: Uploading piece... Dec 27 05:04:43.340545556 (517h35m43.211628591s) [noise=true , address=77.90.31.14:4662 , piece_id=5LZSOGN3TTPQCV2H3H6FZGTUJYEKY24Z6U6VEWUY5WWNCWHMTMZA, seg_pos={0 0}, upload=454119]: Uploading piece... Dec 27 05:04:43.340947532 (517h35m43.212030577s) [noise=true , address=145.79.227.3:4151 , piece_id=64Z4GLFZUWOSR44HPZFTTFTFERA7C3ZST3OXAJEYNQJN6MXROZ5A, seg_pos={0 0}, upload=454119]: Uploading piece... Dec 27 05:04:43.341582400 (517h35m43.212665435s) [noise=true , address=77.90.31.15:4697 , piece_id=5WKWGRREIPFZPTEOOA53XR4IPPD6HDGRIED3DG4VQSV62NKTUJRA, seg_pos={0 0}, upload=454119]: Uploading piece... Dec 27 05:04:43.342355793 (517h35m43.213438828s) [noise=true , address=145.79.227.50:4374 , piece_id=3LL2WIO5F577Z6L6QT4GTNCJOY2FH27E55C752MI2HC5RBIWPUEQ, seg_pos={0 0}, upload=454119]: Uploading piece... Dec 27 05:04:43.342619185 (517h35m43.21370222s) [noise=true , address=45.91.116.5:4297 , piece_id=HK3QXUPBOCCWU6CP246SVF3G56DCUPXL3SN7DHKAEKEI3T4D24QQ, seg_pos={0 0}, upload=454119]: Uploading piece... Dec 27 05:04:43.342988457 (517h35m43.214071492s) [noise=true , address=45.91.116.13:4156 , piece_id=X4B653XODZROZ7K2KCVGJFNDZEZQURUUZDEZVKERXGH54BTXXNPQ, seg_pos={0 0}, upload=454119]: Uploading piece... Dec 27 05:04:43.343368130 (517h35m43.214451165s) [noise=true , address=145.79.227.52:4447 , piece_id=NEPMKF637LZG7UCHVETEPBSHAYHZQLDCJQBI6YGECM3TRJJIQBKQ, seg_pos={0 0}, upload=454119]: Uploading piece... Dec 27 05:04:45.058291195 (517h35m44.92937425s) [upload=454119]: Next returned nil segment Dec 27 05:04:45.058307837 (517h35m44.929390882s) [upload=454119]: Waiting for error group managing segments... Dec 27 05:04:45.348170092 (517h35m45.219253137s) [noise=true , address=145.79.227.23:4546 , piece_id=DT6EHZKNKMTRNCUSVBZPXGCJYLJNJAGKLGVJ62UZZUPDG7YFGAGQ, seg_pos={0 0}, upload=454119]: Done uploading piece. err: <nil> Dec 27 05:04:45.349958328 (517h35m45.221041373s) [noise=true , address=145.79.227.39:4014 , piece_id=SIFRERR4DYHHQJDJY7LROKSOS2EI7DMNTI63NLDG4VYO6HYEOPLA, seg_pos={0 0}, upload=454119]: Done uploading piece. err: <nil> Dec 27 05:04:45.352043719 (517h35m45.223126754s) [noise=true , address=145.79.227.45:4229 , piece_id=LHDM324WTKLUHKJXKR32LOVM4PBEDISYKR3EUVS67W6KO56CSMIQ, seg_pos={0 0}, upload=454119]: Done uploading piece. err: <nil> Dec 27 05:04:45.358988092 (517h35m45.230071127s) [noise=true , address=145.79.227.50:4374 , piece_id=3LL2WIO5F577Z6L6QT4GTNCJOY2FH27E55C752MI2HC5RBIWPUEQ, seg_pos={0 0}, upload=454119]: Done uploading piece. err: <nil> Dec 27 05:04:45.364350012 (517h35m45.235433047s) [noise=true , address=145.79.227.28:4725 , piece_id=4LSPEZDR2XOHZVDRW33D7MIMEPCV5ADDGJJ3WCQ7YGJDWP437H4Q, seg_pos={0 0}, upload=454119]: Done uploading piece. err: <nil> Dec 27 05:04:45.369779811 (517h35m45.240862856s) [noise=true , address=145.79.227.3:4151 , piece_id=64Z4GLFZUWOSR44HPZFTTFTFERA7C3ZST3OXAJEYNQJN6MXROZ5A, seg_pos={0 0}, upload=454119]: Done uploading piece. err: <nil> Dec 27 05:04:45.370780246 (517h35m45.241863291s) [noise=true , address=145.79.227.52:4447 , piece_id=NEPMKF637LZG7UCHVETEPBSHAYHZQLDCJQBI6YGECM3TRJJIQBKQ, seg_pos={0 0}, upload=454119]: Done uploading piece. err: <nil> Dec 27 05:04:45.531586007 (517h35m45.402669052s) [noise=true , address=77.90.31.15:4697 , piece_id=5WKWGRREIPFZPTEOOA53XR4IPPD6HDGRIED3DG4VQSV62NKTUJRA, seg_pos={0 0}, upload=454119]: Done uploading piece. err: <nil> Dec 27 05:04:45.550129095 (517h35m45.42121213s) [noise=true , address=77.90.31.16:4736 , piece_id=BHW4TX3SPKGYQWUYEBM53XTCXEKYGVXBKEX5J6GRQC7G5X5N4PBA, seg_pos={0 0}, upload=454119]: Done uploading piece. err: <nil> Dec 27 05:04:45.565083792 (517h35m45.436166827s) [noise=true , address=45.91.116.8:4045 , piece_id=NNJYU4XCWUPGB6VMEKUU3MOF7YHZG3RYZBTBNOX7QUPKCGED2CTQ, seg_pos={0 0}, upload=454119]: Done uploading piece. err: <nil> Dec 27 05:04:45.671911569 (517h35m45.542994614s) [noise=true , address=45.91.116.5:4297 , piece_id=HK3QXUPBOCCWU6CP246SVF3G56DCUPXL3SN7DHKAEKEI3T4D24QQ, seg_pos={0 0}, upload=454119]: Done uploading piece. err: <nil> Dec 27 05:04:46.427999232 (517h35m46.299082267s) [noise=true , address=45.91.116.4:4940 , piece_id=SEL73K4UYC6EYPU7KTWIYD5B5UBE3ZUBKVEJ4XEKHVAOTSAOANIQ, seg_pos={0 0}, upload=454119]: Done uploading piece. err: <nil> Dec 27 05:04:46.703378850 (517h35m46.574461885s) [noise=true , address=45.91.116.7:4009 , piece_id=QBAZSTRJINY47TBQVOJ2QC3RIBOQ4ULA3EYU4VV2HJ5PYX7UWGGQ, seg_pos={0 0}, upload=454119]: Done uploading piece. err: <nil> Dec 27 05:04:46.703404509 (517h35m46.574487554s) [seg_pos={0 0}, upload=454119]: Segment reached optimal threshold of 13 pieces. Dec 27 05:04:46.703622393 (517h35m46.574705438s) [noise=true , address=45.91.116.13:4156 , piece_id=X4B653XODZROZ7K2KCVGJFNDZEZQURUUZDEZVKERXGH54BTXXNPQ, seg_pos={0 0}, upload=454119]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 05:04:46.703631330 (517h35m46.574714375s) [noise=true , address=45.91.116.13:4156 , piece_id=X4B653XODZROZ7K2KCVGJFNDZEZQURUUZDEZVKERXGH54BTXXNPQ, seg_pos={0 0}, upload=454119]: Long tail cancellation due to optimal threshold Dec 27 05:04:46.703806594 (517h35m46.574889629s) [noise=true , address=77.90.31.14:4662 , piece_id=5LZSOGN3TTPQCV2H3H6FZGTUJYEKY24Z6U6VEWUY5WWNCWHMTMZA, seg_pos={0 0}, upload=454119]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 27 05:04:46.703818227 (517h35m46.574901272s) [noise=true , address=77.90.31.14:4662 , piece_id=5LZSOGN3TTPQCV2H3H6FZGTUJYEKY24Z6U6VEWUY5WWNCWHMTMZA, seg_pos={0 0}, upload=454119]: Long tail cancellation due to optimal threshold Dec 27 05:04:46.703936962 (517h35m46.575020017s) [noise=true , address=145.79.227.33:4871 , piece_id=IVGVIUQI35L2JXVDJW46S5QSX2VRQJAFGQQBQ27IL6KJ2VREECJQ, seg_pos={0 0}, upload=454119]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 27 05:04:46.703946901 (517h35m46.575029946s) [noise=true , address=145.79.227.33:4871 , piece_id=IVGVIUQI35L2JXVDJW46S5QSX2VRQJAFGQQBQ27IL6KJ2VREECJQ, seg_pos={0 0}, upload=454119]: Long tail cancellation due to optimal threshold Dec 27 05:04:46.703972270 (517h35m46.575055315s) [seg_pos={0 0}, upload=454119]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:04:46.704142895 (517h35m46.57522594s) [upload=454119]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:04:46.704150238 (517h35m46.575233283s) [upload=454119]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:04:47.002753728 (517h35m46.873836783s) [upload=454119]: Done uploading segments... Dec 27 05:04:47.002776772 (517h35m46.873859817s) [upload=454119]: Upload finished. err: <nil> Duration: 30.00266118s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 3.480865925s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 05:05:10 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.480897236s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X......................X..................] duration: 46.212359451s trace: https://jaeger.dev.storj.io/trace/60d70ba2fe18218 FAIL (2025-12-27 05:05:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 4 (517h19m18.685148379s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814531152 (517h19m18.685614187s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814852964 (517h19m18.685936009s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815170900 (517h19m18.686253945s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815486661 (517h19m18.686569706s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815835786 (517h19m18.686918831s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.816207564 (517h19m18.687290609s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818225116 (517h19m18.689308151s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818325547 (517h19m18.689408592s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.821132431 (517h19m18.692215466s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.822845684 (517h19m18.693928719s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.824299251 (517h19m18.695382296s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.825754102 (517h19m18.696837147s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.827122548 (517h19m18.698205593s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.828453653 (517h19m18.699536688s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:20.114191076 (517h19m19.985274121s) [upload=453889]: Next returned nil segment Dec 27 04:48:20.114206816 (517h19m19.985289851s) [upload=453889]: Waiting for error group managing segments... Dec 27 04:48:20.444209056 (517h19m20.315292102s) [noise=true , address=64.49.13.154:4929 , piece_id=2EXG7BEQ6NKAG3TNAH3TZ3K5ZI4KNOQYF3OEZMFWVX6BBMWNVXUQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.474597952 (517h19m20.345680987s) [noise=true , address=64.49.13.27:4964 , piece_id=XAROGTY2JLVFUVDYT7N7ZCABVYDMXA45NZF36NBAV4H72M3PJSCA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.478601025 (517h19m20.34968406s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.888413067 (517h19m20.759496102s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.930510167 (517h19m20.801593212s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.007760305 (517h19m20.87884334s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.041631362 (517h19m20.912714407s) [noise=true , address=77.90.31.14:4691 , piece_id=HIFLW7CWEYK6A6BZAYPDZLIUNA26IFXSFNVYV2UK4VPKEWYUMYJA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.052961556 (517h19m20.924044601s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:23.606703638 (517h19m23.477786673s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.046520345 (517h19m25.91760338s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.518771011 (517h19m26.389854046s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.589256778 (517h19m26.460339823s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:27.044280620 (517h19m26.915363655s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789125901 (517h19m36.660208936s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789145728 (517h19m36.660228764s) [seg_pos={0 0}, upload=453889]: Segment reached optimal threshold of 14 pieces. Dec 27 04:48:36.789615964 (517h19m36.660699009s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: read tcp 188.166.225.170:50246->145.79.227.40:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789624180 (517h19m36.660707215s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789761231 (517h19m36.660844276s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: read tcp 188.166.225.170:45976->145.79.227.44:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789766831 (517h19m36.660849866s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789924742 (517h19m36.661007788s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51): protocol: read tcp 188.166.225.170:43134->145.79.227.43:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789933699 (517h19m36.661016734s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790128380 (517h19m36.661211425s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1): protocol: read tcp 188.166.225.170:46462->145.79.227.25:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.790137838 (517h19m36.661220873s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790156633 (517h19m36.661239679s) [seg_pos={0 0}, upload=453889]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:48:36.790241626 (517h19m36.661324661s) [upload=453889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:48:36.790246826 (517h19m36.661329861s) [upload=453889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:48:37.084198870 (517h19m36.955281915s) [upload=453889]: Done uploading segments... Dec 27 04:48:37.084219929 (517h19m36.955302974s) [upload=453889]: Upload finished. err: <nil> Duration: 46.212390271s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.X.X....X...XX.XXX..XXX..XXXX...XX...X.XX....XXXX..X.X.XX..X] duration: 30.004447709s trace: https://jaeger.dev.storj.io/trace/491269e38e641102 FAIL (2025-12-27 05:04:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: 2XP56GBHH73QRIRWRA6RLOMSUKESW7565JGNMFMMBQAC5JCP426A) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: QQ52LCBLCMCTJMSUZ6TDHN6WGWVUXRVUAWHISKXO7HMUONJ2CPOQ) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: ST5P5A2UQJVJ5OAZV45AUWHNRNCQXZ6QIB4YP3M7H3QZF6ZOZCYA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: YNDQSKOMILPIBRLGNPPXBAHCLNOHEI26TWUH2GRPU7TCFT67H6YA) context deadline exceeded ======LOGS====== Dec 27 05:04:12.858232134 (517h35m12.72931517s) [upload=454110]: Starting upload Dec 27 05:04:12.858324079 (517h35m12.729407124s) [upload=454110]: Done starting upload Dec 27 05:04:12.858374025 (517h35m12.72945706s) [upload=454110]: Uploading segments... Dec 27 05:04:12.858776981 (517h35m12.729860026s) [upload=454110]: Got next segment. Inline: false Dec 27 05:04:12.858825635 (517h35m12.72990867s) [upload=454110]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:04:12.858831476 (517h35m12.729914521s) [upload=454110]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:04:13.105455207 (517h35m12.976538252s) [seg_pos={0 0}, upload=454110]: Begin upload segment... Dec 27 05:04:13.105502266 (517h35m12.976585301s) [seg_pos={0 0}, upload=454110]: Done begin upload segment. Dec 27 05:04:13.105532664 (517h35m12.976615709s) [noise=true , address=45.91.116.6:4973 , piece_id=FOUJBHKE7CTCWUDMQU44HFXGZPRJIEHH2ZFLPC56KTCPTGEJVBWQ, seg_pos={0 0}, upload=454110]: Uploading piece... Dec 27 05:04:13.105614229 (517h35m12.976697274s) [noise=true , address=77.90.31.10:4519 , piece_id=YNDQSKOMILPIBRLGNPPXBAHCLNOHEI26TWUH2GRPU7TCFT67H6YA, seg_pos={0 0}, upload=454110]: Uploading piece... Dec 27 05:04:13.106205485 (517h35m12.97728853s) [noise=true , address=145.79.227.51:4410 , piece_id=NPL5Z5M2SHKD7GSUAJ6TCDS62SVPXHIJRSBUQ2HNGCSPI7VMHY5Q, seg_pos={0 0}, upload=454110]: Uploading piece... Dec 27 05:04:13.106283354 (517h35m12.977366389s) [noise=true , address=145.79.227.2:4115 , piece_id=ED6LVMWWJFRTUKLEUHWZSBAMCTTKFYEQSNGSG7R3VH7OFXDKBIVA, seg_pos={0 0}, upload=454110]: Uploading piece... Dec 27 05:04:13.106816038 (517h35m12.977899083s) [noise=true , address=145.79.227.32:4835 , piece_id=4XBTMF3VZXCJZRNZZVRNLRKSUOYXEULDN4KL3J5C3USMIU5CJYXQ, seg_pos={0 0}, upload=454110]: Uploading piece... Dec 27 05:04:13.107244203 (517h35m12.978327248s) [noise=true , address=145.79.227.40:4049 , piece_id=NKH3656QVO6XSXRMFYR3SLV7RQB3ZVDYQWCRVW5XCOBQNHDAQKHQ, seg_pos={0 0}, upload=454110]: Uploading piece... Dec 27 05:04:13.107491754 (517h35m12.978574789s) [noise=true , address=77.90.31.17:4770 , piece_id=2XP56GBHH73QRIRWRA6RLOMSUKESW7565JGNMFMMBQAC5JCP426A, seg_pos={0 0}, upload=454110]: Uploading piece... Dec 27 05:04:13.107548813 (517h35m12.978631858s) [noise=true , address=45.91.116.10:4081 , piece_id=6PYRAOYEJKPVGOADGUZLGNXEEYAUYK7ZEDYZYEXYMENTXRCYNDJA, seg_pos={0 0}, upload=454110]: Uploading piece... Dec 27 05:04:13.107821383 (517h35m12.978904418s) [noise=true , address=45.91.116.14:4191 , piece_id=QQ52LCBLCMCTJMSUZ6TDHN6WGWVUXRVUAWHISKXO7HMUONJ2CPOQ, seg_pos={0 0}, upload=454110]: Uploading piece... Dec 27 05:04:13.108101296 (517h35m12.979184331s) [noise=true , address=77.90.31.12:4589 , piece_id=ST5P5A2UQJVJ5OAZV45AUWHNRNCQXZ6QIB4YP3M7H3QZF6ZOZCYA, seg_pos={0 0}, upload=454110]: Uploading piece... Dec 27 05:04:13.108357824 (517h35m12.979440859s) [noise=true , address=145.79.227.47:4267 , piece_id=TAQ43MVYRTCIRC5KEFANO3IF73SEXT54JSDYSGVE2YFB3TEQUP2Q, seg_pos={0 0}, upload=454110]: Uploading piece... Dec 27 05:04:13.108501698 (517h35m12.979584743s) [noise=true , address=145.79.227.41:4085 , piece_id=FOQRKGAQQJVUGXS7RRGCT5GK6QFTBSRPS7XUQ4RIZKYJ7QQCL37Q, seg_pos={0 0}, upload=454110]: Uploading piece... Dec 27 05:04:13.109122239 (517h35m12.980205284s) [noise=true , address=145.79.227.13:4804 , piece_id=UIPUELSWDHGRTLHF35MLZIWOTWJEF33TA3PBIH5IENZGDKDDYL7A, seg_pos={0 0}, upload=454110]: Uploading piece... Dec 27 05:04:13.109681925 (517h35m12.98076497s) [noise=true , address=145.79.227.16:4911 , piece_id=HLKFXNA4PS62Q36SUOXLUOHSEAOL56H7Y4JAN4XUE6UL33R4FPIQ, seg_pos={0 0}, upload=454110]: Uploading piece... Dec 27 05:04:13.109922714 (517h35m12.981005779s) [noise=true , address=145.79.227.29:4943 , piece_id=AB72U3DDWRSR5BVFEBZ65PJUO4L6EZO5CLJ7U3SUIK2SFLTC34OA, seg_pos={0 0}, upload=454110]: Uploading piece... Dec 27 05:04:13.110364024 (517h35m12.981447069s) [noise=true , address=145.79.227.15:4873 , piece_id=RMPG7HEDG7WVD5LK4SNS3JN5IDAPXQEODZF2JH6LFS4ID6TDXJQA, seg_pos={0 0}, upload=454110]: Uploading piece... Dec 27 05:04:14.825543987 (517h35m14.696627032s) [upload=454110]: Next returned nil segment Dec 27 05:04:14.825558516 (517h35m14.696641551s) [upload=454110]: Waiting for error group managing segments... Dec 27 05:04:15.101590876 (517h35m14.972673911s) [noise=true , address=145.79.227.13:4804 , piece_id=UIPUELSWDHGRTLHF35MLZIWOTWJEF33TA3PBIH5IENZGDKDDYL7A, seg_pos={0 0}, upload=454110]: Done uploading piece. err: <nil> Dec 27 05:04:15.112170742 (517h35m14.983253787s) [noise=true , address=145.79.227.15:4873 , piece_id=RMPG7HEDG7WVD5LK4SNS3JN5IDAPXQEODZF2JH6LFS4ID6TDXJQA, seg_pos={0 0}, upload=454110]: Done uploading piece. err: <nil> Dec 27 05:04:15.114952489 (517h35m14.986035534s) [noise=true , address=145.79.227.51:4410 , piece_id=NPL5Z5M2SHKD7GSUAJ6TCDS62SVPXHIJRSBUQ2HNGCSPI7VMHY5Q, seg_pos={0 0}, upload=454110]: Done uploading piece. err: <nil> Dec 27 05:04:15.116777644 (517h35m14.987860689s) [noise=true , address=145.79.227.16:4911 , piece_id=HLKFXNA4PS62Q36SUOXLUOHSEAOL56H7Y4JAN4XUE6UL33R4FPIQ, seg_pos={0 0}, upload=454110]: Done uploading piece. err: <nil> Dec 27 05:04:15.124489118 (517h35m14.995572153s) [noise=true , address=145.79.227.29:4943 , piece_id=AB72U3DDWRSR5BVFEBZ65PJUO4L6EZO5CLJ7U3SUIK2SFLTC34OA, seg_pos={0 0}, upload=454110]: Done uploading piece. err: <nil> Dec 27 05:04:15.126620877 (517h35m14.997703922s) [noise=true , address=145.79.227.2:4115 , piece_id=ED6LVMWWJFRTUKLEUHWZSBAMCTTKFYEQSNGSG7R3VH7OFXDKBIVA, seg_pos={0 0}, upload=454110]: Done uploading piece. err: <nil> Dec 27 05:04:15.131568638 (517h35m15.002651683s) [noise=true , address=145.79.227.41:4085 , piece_id=FOQRKGAQQJVUGXS7RRGCT5GK6QFTBSRPS7XUQ4RIZKYJ7QQCL37Q, seg_pos={0 0}, upload=454110]: Done uploading piece. err: <nil> Dec 27 05:04:15.136709718 (517h35m15.007792753s) [noise=true , address=145.79.227.32:4835 , piece_id=4XBTMF3VZXCJZRNZZVRNLRKSUOYXEULDN4KL3J5C3USMIU5CJYXQ, seg_pos={0 0}, upload=454110]: Done uploading piece. err: <nil> Dec 27 05:04:15.141187354 (517h35m15.0122704s) [noise=true , address=145.79.227.47:4267 , piece_id=TAQ43MVYRTCIRC5KEFANO3IF73SEXT54JSDYSGVE2YFB3TEQUP2Q, seg_pos={0 0}, upload=454110]: Done uploading piece. err: <nil> Dec 27 05:04:15.279198279 (517h35m15.150281325s) [noise=true , address=77.90.31.10:4519 , piece_id=YNDQSKOMILPIBRLGNPPXBAHCLNOHEI26TWUH2GRPU7TCFT67H6YA, seg_pos={0 0}, upload=454110]: Done uploading piece. err: <nil> Dec 27 05:04:15.356290616 (517h35m15.227373651s) [noise=true , address=45.91.116.14:4191 , piece_id=QQ52LCBLCMCTJMSUZ6TDHN6WGWVUXRVUAWHISKXO7HMUONJ2CPOQ, seg_pos={0 0}, upload=454110]: Done uploading piece. err: <nil> Dec 27 05:04:15.367670545 (517h35m15.23875358s) [noise=true , address=77.90.31.17:4770 , piece_id=2XP56GBHH73QRIRWRA6RLOMSUKESW7565JGNMFMMBQAC5JCP426A, seg_pos={0 0}, upload=454110]: Done uploading piece. err: <nil> Dec 27 05:04:15.394006493 (517h35m15.265089528s) [noise=true , address=77.90.31.12:4589 , piece_id=ST5P5A2UQJVJ5OAZV45AUWHNRNCQXZ6QIB4YP3M7H3QZF6ZOZCYA, seg_pos={0 0}, upload=454110]: Done uploading piece. err: <nil> Dec 27 05:04:15.394031070 (517h35m15.265114115s) [seg_pos={0 0}, upload=454110]: Segment reached optimal threshold of 13 pieces. Dec 27 05:04:15.394289081 (517h35m15.265372126s) [noise=true , address=45.91.116.10:4081 , piece_id=6PYRAOYEJKPVGOADGUZLGNXEEYAUYK7ZEDYZYEXYMENTXRCYNDJA, seg_pos={0 0}, upload=454110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 27 05:04:15.394297327 (517h35m15.265380362s) [noise=true , address=45.91.116.10:4081 , piece_id=6PYRAOYEJKPVGOADGUZLGNXEEYAUYK7ZEDYZYEXYMENTXRCYNDJA, seg_pos={0 0}, upload=454110]: Long tail cancellation due to optimal threshold Dec 27 05:04:15.394673072 (517h35m15.265756107s) [noise=true , address=45.91.116.6:4973 , piece_id=FOUJBHKE7CTCWUDMQU44HFXGZPRJIEHH2ZFLPC56KTCPTGEJVBWQ, seg_pos={0 0}, upload=454110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 27 05:04:15.394691337 (517h35m15.265774383s) [noise=true , address=45.91.116.6:4973 , piece_id=FOUJBHKE7CTCWUDMQU44HFXGZPRJIEHH2ZFLPC56KTCPTGEJVBWQ, seg_pos={0 0}, upload=454110]: Long tail cancellation due to optimal threshold Dec 27 05:04:15.394763064 (517h35m15.265846099s) [noise=true , address=145.79.227.40:4049 , piece_id=NKH3656QVO6XSXRMFYR3SLV7RQB3ZVDYQWCRVW5XCOBQNHDAQKHQ, seg_pos={0 0}, upload=454110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 27 05:04:15.394771760 (517h35m15.265854795s) [noise=true , address=145.79.227.40:4049 , piece_id=NKH3656QVO6XSXRMFYR3SLV7RQB3ZVDYQWCRVW5XCOBQNHDAQKHQ, seg_pos={0 0}, upload=454110]: Long tail cancellation due to optimal threshold Dec 27 05:04:15.394794354 (517h35m15.265877399s) [seg_pos={0 0}, upload=454110]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:04:15.394914752 (517h35m15.265997788s) [upload=454110]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:04:15.394922066 (517h35m15.266005101s) [upload=454110]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:04:15.697076068 (517h35m15.568159103s) [upload=454110]: Done uploading segments... Dec 27 05:04:15.697096556 (517h35m15.568179591s) [upload=454110]: Upload finished. err: <nil> Duration: 30.004471534s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 3.591680464s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 05:04:41 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.591706783s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 3.68449967s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 05:04:20 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.684549426s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X......................X.................] duration: 48.754439261s trace: https://jaeger.dev.storj.io/trace/60d70ba2fe18218 FAIL (2025-12-27 05:04:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 4 (517h19m18.685148379s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814531152 (517h19m18.685614187s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814852964 (517h19m18.685936009s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815170900 (517h19m18.686253945s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815486661 (517h19m18.686569706s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815835786 (517h19m18.686918831s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.816207564 (517h19m18.687290609s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818225116 (517h19m18.689308151s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818325547 (517h19m18.689408592s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.821132431 (517h19m18.692215466s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.822845684 (517h19m18.693928719s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.824299251 (517h19m18.695382296s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.825754102 (517h19m18.696837147s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.827122548 (517h19m18.698205593s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.828453653 (517h19m18.699536688s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:20.114191076 (517h19m19.985274121s) [upload=453889]: Next returned nil segment Dec 27 04:48:20.114206816 (517h19m19.985289851s) [upload=453889]: Waiting for error group managing segments... Dec 27 04:48:20.444209056 (517h19m20.315292102s) [noise=true , address=64.49.13.154:4929 , piece_id=2EXG7BEQ6NKAG3TNAH3TZ3K5ZI4KNOQYF3OEZMFWVX6BBMWNVXUQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.474597952 (517h19m20.345680987s) [noise=true , address=64.49.13.27:4964 , piece_id=XAROGTY2JLVFUVDYT7N7ZCABVYDMXA45NZF36NBAV4H72M3PJSCA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.478601025 (517h19m20.34968406s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.888413067 (517h19m20.759496102s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.930510167 (517h19m20.801593212s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.007760305 (517h19m20.87884334s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.041631362 (517h19m20.912714407s) [noise=true , address=77.90.31.14:4691 , piece_id=HIFLW7CWEYK6A6BZAYPDZLIUNA26IFXSFNVYV2UK4VPKEWYUMYJA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.052961556 (517h19m20.924044601s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:23.606703638 (517h19m23.477786673s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.046520345 (517h19m25.91760338s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.518771011 (517h19m26.389854046s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.589256778 (517h19m26.460339823s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:27.044280620 (517h19m26.915363655s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789125901 (517h19m36.660208936s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789145728 (517h19m36.660228764s) [seg_pos={0 0}, upload=453889]: Segment reached optimal threshold of 14 pieces. Dec 27 04:48:36.789615964 (517h19m36.660699009s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: read tcp 188.166.225.170:50246->145.79.227.40:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789624180 (517h19m36.660707215s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789761231 (517h19m36.660844276s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: read tcp 188.166.225.170:45976->145.79.227.44:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789766831 (517h19m36.660849866s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789924742 (517h19m36.661007788s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51): protocol: read tcp 188.166.225.170:43134->145.79.227.43:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789933699 (517h19m36.661016734s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790128380 (517h19m36.661211425s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1): protocol: read tcp 188.166.225.170:46462->145.79.227.25:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.790137838 (517h19m36.661220873s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790156633 (517h19m36.661239679s) [seg_pos={0 0}, upload=453889]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:48:36.790241626 (517h19m36.661324661s) [upload=453889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:48:36.790246826 (517h19m36.661329861s) [upload=453889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:48:37.084198870 (517h19m36.955281915s) [upload=453889]: Done uploading segments... Dec 27 04:48:37.084219929 (517h19m36.955302974s) [upload=453889]: Upload finished. err: <nil> Duration: 48.754467827s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X.X.X....X...XX.XXX..XXX..XXXX...XX...X.XX....XXXX..X.X.XX..] duration: 6.114053026s trace: https://jaeger.dev.storj.io/trace/5a204fbd1d5aacd9 FAIL (2025-12-27 05:03:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: DSJA5UGGUE322S5MOEOH27DNM4PPCB2BJ4F7CRALKOYJZWQBMGCQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: VLPOQHQ2GFEJ7L7TG5ZEYAKBMCI65Z6HP2VCFYUZ7E5RJOKYWXXA) context deadline exceeded ======LOGS====== Dec 27 05:02:13.477434277 (517h33m13.348517322s) [upload=454082]: Starting upload Dec 27 05:02:13.477540299 (517h33m13.348623334s) [upload=454082]: Done starting upload Dec 27 05:02:13.477647163 (517h33m13.348730208s) [upload=454082]: Uploading segments... Dec 27 05:02:13.477793852 (517h33m13.348876887s) [upload=454082]: Got next segment. Inline: false Dec 27 05:02:13.477841513 (517h33m13.348924548s) [upload=454082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:02:13.477847594 (517h33m13.34893064s) [upload=454082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:02:13.724126840 (517h33m13.595209875s) [seg_pos={0 0}, upload=454082]: Begin upload segment... Dec 27 05:02:13.724178939 (517h33m13.595261974s) [seg_pos={0 0}, upload=454082]: Done begin upload segment. Dec 27 05:02:13.724221069 (517h33m13.595304104s) [noise=true , address=45.91.116.4:4937 , piece_id=T45QMUIHYFAH7LOHIBKF5P5UMTIHBYL3IX2R7L3BMZMMC5YDWOYQ, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.724255935 (517h33m13.59533897s) [noise=true , address=145.79.227.48:4302 , piece_id=MJEEFXBNQ2GDP6FC4354AUFMZ7ZNQRTA3UZKIXSPLYHRNPDB4YNA, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725030070 (517h33m13.596113125s) [noise=true , address=145.79.227.36:4518 , piece_id=UEX7HJ7JOGF36245ZLXJMCKCNT3HTAXTANA5N3YJ4FGNJVMGIJMQ, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725163483 (517h33m13.596246528s) [noise=true , address=145.79.227.46:4602 , piece_id=W24GFKWDSLV2TWYIAX2G4MYZTXOGKVNVALM7FOYSKZ5MMZDLFEAQ, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725234720 (517h33m13.596317765s) [noise=true , address=145.79.227.49:4337 , piece_id=F2R4QSOXQELSQRD6SJYCCI7ECN7GX5TPXIYKVCG6EQDWDLF3RAAQ, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725285085 (517h33m13.59636812s) [noise=true , address=145.79.227.52:4446 , piece_id=WVXCET4VDJEN5GXCYLHDGCWT7KQDUBJPRMTY4XDSHR3MINDIBQRQ, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725315894 (517h33m13.596398929s) [noise=true , address=145.79.227.11:4729 , piece_id=UJWE4UOAWWEJY43LP3I5R6XUB3E2AKQCK2KSAITJQPDRRXTP5A6A, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725348276 (517h33m13.596431321s) [noise=true , address=45.91.116.10:4082 , piece_id=QZQVUQO7YVHBI4ZNPGFCFSD3SUQUI2GHTQTGSMNYEJLY6T2HBZJA, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725377822 (517h33m13.596460867s) [noise=true , address=145.79.227.12:4765 , piece_id=CUEYAIJF2XLUZQU6Q33ZXNHQ7OOBSMKJ2ZLQMB6BIYALRAO6FG2Q, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725412718 (517h33m13.596495754s) [noise=true , address=45.91.116.9:4262 , piece_id=DSJA5UGGUE322S5MOEOH27DNM4PPCB2BJ4F7CRALKOYJZWQBMGCQ, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725438347 (517h33m13.596521382s) [noise=true , address=145.79.227.37:4566 , piece_id=ECCVABXFVUH3LHMDR4KC35M3PZT7G3QKKXG7GGA6J5LXI2INFQEA, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725465278 (517h33m13.596548313s) [noise=true , address=145.79.227.39:4013 , piece_id=RSRMT53QWCG2YYWNIK6ODEQ6LAIZDKXFDPIJELD4ETBIHALJSDZA, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.726370642 (517h33m13.597453677s) [noise=true , address=145.79.227.38:4978 , piece_id=BYOEMVXFDGNYWBTTDH2KND6G4OSXDMQ3GHZLFIOKSQQYZ5AHFITA, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.727444086 (517h33m13.598527121s) [noise=true , address=145.79.227.51:4409 , piece_id=K5TPWU3XHKQKFB5ZNQ5MRE7MGZEBQ5DGLFLIUJZS5UAHMMYBEIUA, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.727700244 (517h33m13.598783309s) [noise=true , address=77.90.31.19:4841 , piece_id=VLPOQHQ2GFEJ7L7TG5ZEYAKBMCI65Z6HP2VCFYUZ7E5RJOKYWXXA, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.728594026 (517h33m13.599677071s) [noise=true , address=45.91.116.6:4976 , piece_id=YAWNNDCLDBY6ZPEWGHTZK22SWYBDVL7IXULMJALJTOHOEBGOL45Q, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:15.442938397 (517h33m15.314021432s) [upload=454082]: Next returned nil segment Dec 27 05:02:15.442957853 (517h33m15.314040899s) [upload=454082]: Waiting for error group managing segments... Dec 27 05:02:15.728204690 (517h33m15.599287726s) [noise=true , address=145.79.227.49:4337 , piece_id=F2R4QSOXQELSQRD6SJYCCI7ECN7GX5TPXIYKVCG6EQDWDLF3RAAQ, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.729084425 (517h33m15.60016747s) [noise=true , address=145.79.227.38:4978 , piece_id=BYOEMVXFDGNYWBTTDH2KND6G4OSXDMQ3GHZLFIOKSQQYZ5AHFITA, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.732214646 (517h33m15.603297681s) [noise=true , address=145.79.227.36:4518 , piece_id=UEX7HJ7JOGF36245ZLXJMCKCNT3HTAXTANA5N3YJ4FGNJVMGIJMQ, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.735941773 (517h33m15.607024818s) [noise=true , address=145.79.227.48:4302 , piece_id=MJEEFXBNQ2GDP6FC4354AUFMZ7ZNQRTA3UZKIXSPLYHRNPDB4YNA, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.736777655 (517h33m15.6078607s) [noise=true , address=145.79.227.11:4729 , piece_id=UJWE4UOAWWEJY43LP3I5R6XUB3E2AKQCK2KSAITJQPDRRXTP5A6A, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.751386853 (517h33m15.622469898s) [noise=true , address=145.79.227.52:4446 , piece_id=WVXCET4VDJEN5GXCYLHDGCWT7KQDUBJPRMTY4XDSHR3MINDIBQRQ, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.751453440 (517h33m15.622536475s) [noise=true , address=145.79.227.46:4602 , piece_id=W24GFKWDSLV2TWYIAX2G4MYZTXOGKVNVALM7FOYSKZ5MMZDLFEAQ, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.751533623 (517h33m15.622616668s) [noise=true , address=145.79.227.12:4765 , piece_id=CUEYAIJF2XLUZQU6Q33ZXNHQ7OOBSMKJ2ZLQMB6BIYALRAO6FG2Q, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.752806868 (517h33m15.623889913s) [noise=true , address=145.79.227.51:4409 , piece_id=K5TPWU3XHKQKFB5ZNQ5MRE7MGZEBQ5DGLFLIUJZS5UAHMMYBEIUA, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.772468056 (517h33m15.643551091s) [noise=true , address=145.79.227.37:4566 , piece_id=ECCVABXFVUH3LHMDR4KC35M3PZT7G3QKKXG7GGA6J5LXI2INFQEA, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.774243326 (517h33m15.645326371s) [noise=true , address=145.79.227.39:4013 , piece_id=RSRMT53QWCG2YYWNIK6ODEQ6LAIZDKXFDPIJELD4ETBIHALJSDZA, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.996738864 (517h33m15.867821909s) [noise=true , address=45.91.116.9:4262 , piece_id=DSJA5UGGUE322S5MOEOH27DNM4PPCB2BJ4F7CRALKOYJZWQBMGCQ, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:16.007683674 (517h33m15.878766719s) [noise=true , address=77.90.31.19:4841 , piece_id=VLPOQHQ2GFEJ7L7TG5ZEYAKBMCI65Z6HP2VCFYUZ7E5RJOKYWXXA, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:16.007709213 (517h33m15.878792248s) [seg_pos={0 0}, upload=454082]: Segment reached optimal threshold of 13 pieces. Dec 27 05:02:16.007987132 (517h33m15.879070187s) [noise=true , address=45.91.116.10:4082 , piece_id=QZQVUQO7YVHBI4ZNPGFCFSD3SUQUI2GHTQTGSMNYEJLY6T2HBZJA, seg_pos={0 0}, upload=454082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 27 05:02:16.007998333 (517h33m15.879081378s) [noise=true , address=45.91.116.10:4082 , piece_id=QZQVUQO7YVHBI4ZNPGFCFSD3SUQUI2GHTQTGSMNYEJLY6T2HBZJA, seg_pos={0 0}, upload=454082]: Long tail cancellation due to optimal threshold Dec 27 05:02:16.008167115 (517h33m15.87925015s) [noise=true , address=45.91.116.4:4937 , piece_id=T45QMUIHYFAH7LOHIBKF5P5UMTIHBYL3IX2R7L3BMZMMC5YDWOYQ, seg_pos={0 0}, upload=454082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 27 05:02:16.008173888 (517h33m15.879256923s) [noise=true , address=45.91.116.4:4937 , piece_id=T45QMUIHYFAH7LOHIBKF5P5UMTIHBYL3IX2R7L3BMZMMC5YDWOYQ, seg_pos={0 0}, upload=454082]: Long tail cancellation due to optimal threshold Dec 27 05:02:16.008240494 (517h33m15.87932357s) [noise=true , address=45.91.116.6:4976 , piece_id=YAWNNDCLDBY6ZPEWGHTZK22SWYBDVL7IXULMJALJTOHOEBGOL45Q, seg_pos={0 0}, upload=454082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:42760->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:42760->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42760->45.91.116.6:4976: use of closed network connection Dec 27 05:02:16.008267045 (517h33m15.87935011s) [noise=true , address=45.91.116.6:4976 , piece_id=YAWNNDCLDBY6ZPEWGHTZK22SWYBDVL7IXULMJALJTOHOEBGOL45Q, seg_pos={0 0}, upload=454082]: Long tail cancellation due to optimal threshold Dec 27 05:02:16.008305849 (517h33m15.879388894s) [seg_pos={0 0}, upload=454082]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:02:16.008411851 (517h33m15.879494896s) [upload=454082]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:02:16.008418193 (517h33m15.879501228s) [upload=454082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:02:16.300259997 (517h33m16.171343042s) [upload=454082]: Done uploading segments... Dec 27 05:02:16.300283272 (517h33m16.171366307s) [upload=454082]: Upload finished. err: <nil> Duration: 6.114084035s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 3.61466365s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 05:03:40 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.614687375s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 3.994754314s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 05:03:07 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.994803828s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X......................X................] duration: 39.423969024s trace: https://jaeger.dev.storj.io/trace/60d70ba2fe18218 FAIL (2025-12-27 05:03:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 4 (517h19m18.685148379s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814531152 (517h19m18.685614187s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814852964 (517h19m18.685936009s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815170900 (517h19m18.686253945s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815486661 (517h19m18.686569706s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815835786 (517h19m18.686918831s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.816207564 (517h19m18.687290609s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818225116 (517h19m18.689308151s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818325547 (517h19m18.689408592s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.821132431 (517h19m18.692215466s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.822845684 (517h19m18.693928719s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.824299251 (517h19m18.695382296s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.825754102 (517h19m18.696837147s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.827122548 (517h19m18.698205593s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.828453653 (517h19m18.699536688s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:20.114191076 (517h19m19.985274121s) [upload=453889]: Next returned nil segment Dec 27 04:48:20.114206816 (517h19m19.985289851s) [upload=453889]: Waiting for error group managing segments... Dec 27 04:48:20.444209056 (517h19m20.315292102s) [noise=true , address=64.49.13.154:4929 , piece_id=2EXG7BEQ6NKAG3TNAH3TZ3K5ZI4KNOQYF3OEZMFWVX6BBMWNVXUQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.474597952 (517h19m20.345680987s) [noise=true , address=64.49.13.27:4964 , piece_id=XAROGTY2JLVFUVDYT7N7ZCABVYDMXA45NZF36NBAV4H72M3PJSCA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.478601025 (517h19m20.34968406s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.888413067 (517h19m20.759496102s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.930510167 (517h19m20.801593212s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.007760305 (517h19m20.87884334s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.041631362 (517h19m20.912714407s) [noise=true , address=77.90.31.14:4691 , piece_id=HIFLW7CWEYK6A6BZAYPDZLIUNA26IFXSFNVYV2UK4VPKEWYUMYJA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.052961556 (517h19m20.924044601s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:23.606703638 (517h19m23.477786673s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.046520345 (517h19m25.91760338s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.518771011 (517h19m26.389854046s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.589256778 (517h19m26.460339823s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:27.044280620 (517h19m26.915363655s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789125901 (517h19m36.660208936s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789145728 (517h19m36.660228764s) [seg_pos={0 0}, upload=453889]: Segment reached optimal threshold of 14 pieces. Dec 27 04:48:36.789615964 (517h19m36.660699009s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: read tcp 188.166.225.170:50246->145.79.227.40:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789624180 (517h19m36.660707215s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789761231 (517h19m36.660844276s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: read tcp 188.166.225.170:45976->145.79.227.44:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789766831 (517h19m36.660849866s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789924742 (517h19m36.661007788s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51): protocol: read tcp 188.166.225.170:43134->145.79.227.43:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789933699 (517h19m36.661016734s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790128380 (517h19m36.661211425s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1): protocol: read tcp 188.166.225.170:46462->145.79.227.25:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.790137838 (517h19m36.661220873s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790156633 (517h19m36.661239679s) [seg_pos={0 0}, upload=453889]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:48:36.790241626 (517h19m36.661324661s) [upload=453889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:48:36.790246826 (517h19m36.661329861s) [upload=453889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:48:37.084198870 (517h19m36.955281915s) [upload=453889]: Done uploading segments... Dec 27 04:48:37.084219929 (517h19m36.955302974s) [upload=453889]: Upload finished. err: <nil> Duration: 39.423986367s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [XX.X.X....X...XX.XXX..XXX..XXXX...XX...X.XX....XXXX..X.X.XX.] duration: 6.0539541s trace: https://jaeger.dev.storj.io/trace/5a204fbd1d5aacd9 FAIL (2025-12-27 05:02:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: DSJA5UGGUE322S5MOEOH27DNM4PPCB2BJ4F7CRALKOYJZWQBMGCQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: VLPOQHQ2GFEJ7L7TG5ZEYAKBMCI65Z6HP2VCFYUZ7E5RJOKYWXXA) context deadline exceeded ======LOGS====== Dec 27 05:02:13.477434277 (517h33m13.348517322s) [upload=454082]: Starting upload Dec 27 05:02:13.477540299 (517h33m13.348623334s) [upload=454082]: Done starting upload Dec 27 05:02:13.477647163 (517h33m13.348730208s) [upload=454082]: Uploading segments... Dec 27 05:02:13.477793852 (517h33m13.348876887s) [upload=454082]: Got next segment. Inline: false Dec 27 05:02:13.477841513 (517h33m13.348924548s) [upload=454082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:02:13.477847594 (517h33m13.34893064s) [upload=454082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:02:13.724126840 (517h33m13.595209875s) [seg_pos={0 0}, upload=454082]: Begin upload segment... Dec 27 05:02:13.724178939 (517h33m13.595261974s) [seg_pos={0 0}, upload=454082]: Done begin upload segment. Dec 27 05:02:13.724221069 (517h33m13.595304104s) [noise=true , address=45.91.116.4:4937 , piece_id=T45QMUIHYFAH7LOHIBKF5P5UMTIHBYL3IX2R7L3BMZMMC5YDWOYQ, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.724255935 (517h33m13.59533897s) [noise=true , address=145.79.227.48:4302 , piece_id=MJEEFXBNQ2GDP6FC4354AUFMZ7ZNQRTA3UZKIXSPLYHRNPDB4YNA, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725030070 (517h33m13.596113125s) [noise=true , address=145.79.227.36:4518 , piece_id=UEX7HJ7JOGF36245ZLXJMCKCNT3HTAXTANA5N3YJ4FGNJVMGIJMQ, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725163483 (517h33m13.596246528s) [noise=true , address=145.79.227.46:4602 , piece_id=W24GFKWDSLV2TWYIAX2G4MYZTXOGKVNVALM7FOYSKZ5MMZDLFEAQ, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725234720 (517h33m13.596317765s) [noise=true , address=145.79.227.49:4337 , piece_id=F2R4QSOXQELSQRD6SJYCCI7ECN7GX5TPXIYKVCG6EQDWDLF3RAAQ, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725285085 (517h33m13.59636812s) [noise=true , address=145.79.227.52:4446 , piece_id=WVXCET4VDJEN5GXCYLHDGCWT7KQDUBJPRMTY4XDSHR3MINDIBQRQ, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725315894 (517h33m13.596398929s) [noise=true , address=145.79.227.11:4729 , piece_id=UJWE4UOAWWEJY43LP3I5R6XUB3E2AKQCK2KSAITJQPDRRXTP5A6A, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725348276 (517h33m13.596431321s) [noise=true , address=45.91.116.10:4082 , piece_id=QZQVUQO7YVHBI4ZNPGFCFSD3SUQUI2GHTQTGSMNYEJLY6T2HBZJA, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725377822 (517h33m13.596460867s) [noise=true , address=145.79.227.12:4765 , piece_id=CUEYAIJF2XLUZQU6Q33ZXNHQ7OOBSMKJ2ZLQMB6BIYALRAO6FG2Q, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725412718 (517h33m13.596495754s) [noise=true , address=45.91.116.9:4262 , piece_id=DSJA5UGGUE322S5MOEOH27DNM4PPCB2BJ4F7CRALKOYJZWQBMGCQ, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725438347 (517h33m13.596521382s) [noise=true , address=145.79.227.37:4566 , piece_id=ECCVABXFVUH3LHMDR4KC35M3PZT7G3QKKXG7GGA6J5LXI2INFQEA, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725465278 (517h33m13.596548313s) [noise=true , address=145.79.227.39:4013 , piece_id=RSRMT53QWCG2YYWNIK6ODEQ6LAIZDKXFDPIJELD4ETBIHALJSDZA, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.726370642 (517h33m13.597453677s) [noise=true , address=145.79.227.38:4978 , piece_id=BYOEMVXFDGNYWBTTDH2KND6G4OSXDMQ3GHZLFIOKSQQYZ5AHFITA, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.727444086 (517h33m13.598527121s) [noise=true , address=145.79.227.51:4409 , piece_id=K5TPWU3XHKQKFB5ZNQ5MRE7MGZEBQ5DGLFLIUJZS5UAHMMYBEIUA, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.727700244 (517h33m13.598783309s) [noise=true , address=77.90.31.19:4841 , piece_id=VLPOQHQ2GFEJ7L7TG5ZEYAKBMCI65Z6HP2VCFYUZ7E5RJOKYWXXA, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.728594026 (517h33m13.599677071s) [noise=true , address=45.91.116.6:4976 , piece_id=YAWNNDCLDBY6ZPEWGHTZK22SWYBDVL7IXULMJALJTOHOEBGOL45Q, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:15.442938397 (517h33m15.314021432s) [upload=454082]: Next returned nil segment Dec 27 05:02:15.442957853 (517h33m15.314040899s) [upload=454082]: Waiting for error group managing segments... Dec 27 05:02:15.728204690 (517h33m15.599287726s) [noise=true , address=145.79.227.49:4337 , piece_id=F2R4QSOXQELSQRD6SJYCCI7ECN7GX5TPXIYKVCG6EQDWDLF3RAAQ, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.729084425 (517h33m15.60016747s) [noise=true , address=145.79.227.38:4978 , piece_id=BYOEMVXFDGNYWBTTDH2KND6G4OSXDMQ3GHZLFIOKSQQYZ5AHFITA, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.732214646 (517h33m15.603297681s) [noise=true , address=145.79.227.36:4518 , piece_id=UEX7HJ7JOGF36245ZLXJMCKCNT3HTAXTANA5N3YJ4FGNJVMGIJMQ, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.735941773 (517h33m15.607024818s) [noise=true , address=145.79.227.48:4302 , piece_id=MJEEFXBNQ2GDP6FC4354AUFMZ7ZNQRTA3UZKIXSPLYHRNPDB4YNA, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.736777655 (517h33m15.6078607s) [noise=true , address=145.79.227.11:4729 , piece_id=UJWE4UOAWWEJY43LP3I5R6XUB3E2AKQCK2KSAITJQPDRRXTP5A6A, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.751386853 (517h33m15.622469898s) [noise=true , address=145.79.227.52:4446 , piece_id=WVXCET4VDJEN5GXCYLHDGCWT7KQDUBJPRMTY4XDSHR3MINDIBQRQ, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.751453440 (517h33m15.622536475s) [noise=true , address=145.79.227.46:4602 , piece_id=W24GFKWDSLV2TWYIAX2G4MYZTXOGKVNVALM7FOYSKZ5MMZDLFEAQ, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.751533623 (517h33m15.622616668s) [noise=true , address=145.79.227.12:4765 , piece_id=CUEYAIJF2XLUZQU6Q33ZXNHQ7OOBSMKJ2ZLQMB6BIYALRAO6FG2Q, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.752806868 (517h33m15.623889913s) [noise=true , address=145.79.227.51:4409 , piece_id=K5TPWU3XHKQKFB5ZNQ5MRE7MGZEBQ5DGLFLIUJZS5UAHMMYBEIUA, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.772468056 (517h33m15.643551091s) [noise=true , address=145.79.227.37:4566 , piece_id=ECCVABXFVUH3LHMDR4KC35M3PZT7G3QKKXG7GGA6J5LXI2INFQEA, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.774243326 (517h33m15.645326371s) [noise=true , address=145.79.227.39:4013 , piece_id=RSRMT53QWCG2YYWNIK6ODEQ6LAIZDKXFDPIJELD4ETBIHALJSDZA, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.996738864 (517h33m15.867821909s) [noise=true , address=45.91.116.9:4262 , piece_id=DSJA5UGGUE322S5MOEOH27DNM4PPCB2BJ4F7CRALKOYJZWQBMGCQ, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:16.007683674 (517h33m15.878766719s) [noise=true , address=77.90.31.19:4841 , piece_id=VLPOQHQ2GFEJ7L7TG5ZEYAKBMCI65Z6HP2VCFYUZ7E5RJOKYWXXA, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:16.007709213 (517h33m15.878792248s) [seg_pos={0 0}, upload=454082]: Segment reached optimal threshold of 13 pieces. Dec 27 05:02:16.007987132 (517h33m15.879070187s) [noise=true , address=45.91.116.10:4082 , piece_id=QZQVUQO7YVHBI4ZNPGFCFSD3SUQUI2GHTQTGSMNYEJLY6T2HBZJA, seg_pos={0 0}, upload=454082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 27 05:02:16.007998333 (517h33m15.879081378s) [noise=true , address=45.91.116.10:4082 , piece_id=QZQVUQO7YVHBI4ZNPGFCFSD3SUQUI2GHTQTGSMNYEJLY6T2HBZJA, seg_pos={0 0}, upload=454082]: Long tail cancellation due to optimal threshold Dec 27 05:02:16.008167115 (517h33m15.87925015s) [noise=true , address=45.91.116.4:4937 , piece_id=T45QMUIHYFAH7LOHIBKF5P5UMTIHBYL3IX2R7L3BMZMMC5YDWOYQ, seg_pos={0 0}, upload=454082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 27 05:02:16.008173888 (517h33m15.879256923s) [noise=true , address=45.91.116.4:4937 , piece_id=T45QMUIHYFAH7LOHIBKF5P5UMTIHBYL3IX2R7L3BMZMMC5YDWOYQ, seg_pos={0 0}, upload=454082]: Long tail cancellation due to optimal threshold Dec 27 05:02:16.008240494 (517h33m15.87932357s) [noise=true , address=45.91.116.6:4976 , piece_id=YAWNNDCLDBY6ZPEWGHTZK22SWYBDVL7IXULMJALJTOHOEBGOL45Q, seg_pos={0 0}, upload=454082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:42760->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:42760->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42760->45.91.116.6:4976: use of closed network connection Dec 27 05:02:16.008267045 (517h33m15.87935011s) [noise=true , address=45.91.116.6:4976 , piece_id=YAWNNDCLDBY6ZPEWGHTZK22SWYBDVL7IXULMJALJTOHOEBGOL45Q, seg_pos={0 0}, upload=454082]: Long tail cancellation due to optimal threshold Dec 27 05:02:16.008305849 (517h33m15.879388894s) [seg_pos={0 0}, upload=454082]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:02:16.008411851 (517h33m15.879494896s) [upload=454082]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:02:16.008418193 (517h33m15.879501228s) [upload=454082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:02:16.300259997 (517h33m16.171343042s) [upload=454082]: Done uploading segments... Dec 27 05:02:16.300283272 (517h33m16.171366307s) [upload=454082]: Upload finished. err: <nil> Duration: 6.05398096s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [XXX.X.X....X...XX.XXX..XXX..XXXX...XX...X.XX....XXXX..X.X.XX] duration: 30.002770516s trace: https://jaeger.dev.storj.io/trace/5a204fbd1d5aacd9 FAIL (2025-12-27 05:02:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: DSJA5UGGUE322S5MOEOH27DNM4PPCB2BJ4F7CRALKOYJZWQBMGCQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: VLPOQHQ2GFEJ7L7TG5ZEYAKBMCI65Z6HP2VCFYUZ7E5RJOKYWXXA) context deadline exceeded ======LOGS====== Dec 27 05:02:13.477434277 (517h33m13.348517322s) [upload=454082]: Starting upload Dec 27 05:02:13.477540299 (517h33m13.348623334s) [upload=454082]: Done starting upload Dec 27 05:02:13.477647163 (517h33m13.348730208s) [upload=454082]: Uploading segments... Dec 27 05:02:13.477793852 (517h33m13.348876887s) [upload=454082]: Got next segment. Inline: false Dec 27 05:02:13.477841513 (517h33m13.348924548s) [upload=454082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:02:13.477847594 (517h33m13.34893064s) [upload=454082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:02:13.724126840 (517h33m13.595209875s) [seg_pos={0 0}, upload=454082]: Begin upload segment... Dec 27 05:02:13.724178939 (517h33m13.595261974s) [seg_pos={0 0}, upload=454082]: Done begin upload segment. Dec 27 05:02:13.724221069 (517h33m13.595304104s) [noise=true , address=45.91.116.4:4937 , piece_id=T45QMUIHYFAH7LOHIBKF5P5UMTIHBYL3IX2R7L3BMZMMC5YDWOYQ, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.724255935 (517h33m13.59533897s) [noise=true , address=145.79.227.48:4302 , piece_id=MJEEFXBNQ2GDP6FC4354AUFMZ7ZNQRTA3UZKIXSPLYHRNPDB4YNA, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725030070 (517h33m13.596113125s) [noise=true , address=145.79.227.36:4518 , piece_id=UEX7HJ7JOGF36245ZLXJMCKCNT3HTAXTANA5N3YJ4FGNJVMGIJMQ, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725163483 (517h33m13.596246528s) [noise=true , address=145.79.227.46:4602 , piece_id=W24GFKWDSLV2TWYIAX2G4MYZTXOGKVNVALM7FOYSKZ5MMZDLFEAQ, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725234720 (517h33m13.596317765s) [noise=true , address=145.79.227.49:4337 , piece_id=F2R4QSOXQELSQRD6SJYCCI7ECN7GX5TPXIYKVCG6EQDWDLF3RAAQ, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725285085 (517h33m13.59636812s) [noise=true , address=145.79.227.52:4446 , piece_id=WVXCET4VDJEN5GXCYLHDGCWT7KQDUBJPRMTY4XDSHR3MINDIBQRQ, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725315894 (517h33m13.596398929s) [noise=true , address=145.79.227.11:4729 , piece_id=UJWE4UOAWWEJY43LP3I5R6XUB3E2AKQCK2KSAITJQPDRRXTP5A6A, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725348276 (517h33m13.596431321s) [noise=true , address=45.91.116.10:4082 , piece_id=QZQVUQO7YVHBI4ZNPGFCFSD3SUQUI2GHTQTGSMNYEJLY6T2HBZJA, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725377822 (517h33m13.596460867s) [noise=true , address=145.79.227.12:4765 , piece_id=CUEYAIJF2XLUZQU6Q33ZXNHQ7OOBSMKJ2ZLQMB6BIYALRAO6FG2Q, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725412718 (517h33m13.596495754s) [noise=true , address=45.91.116.9:4262 , piece_id=DSJA5UGGUE322S5MOEOH27DNM4PPCB2BJ4F7CRALKOYJZWQBMGCQ, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725438347 (517h33m13.596521382s) [noise=true , address=145.79.227.37:4566 , piece_id=ECCVABXFVUH3LHMDR4KC35M3PZT7G3QKKXG7GGA6J5LXI2INFQEA, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.725465278 (517h33m13.596548313s) [noise=true , address=145.79.227.39:4013 , piece_id=RSRMT53QWCG2YYWNIK6ODEQ6LAIZDKXFDPIJELD4ETBIHALJSDZA, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.726370642 (517h33m13.597453677s) [noise=true , address=145.79.227.38:4978 , piece_id=BYOEMVXFDGNYWBTTDH2KND6G4OSXDMQ3GHZLFIOKSQQYZ5AHFITA, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.727444086 (517h33m13.598527121s) [noise=true , address=145.79.227.51:4409 , piece_id=K5TPWU3XHKQKFB5ZNQ5MRE7MGZEBQ5DGLFLIUJZS5UAHMMYBEIUA, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.727700244 (517h33m13.598783309s) [noise=true , address=77.90.31.19:4841 , piece_id=VLPOQHQ2GFEJ7L7TG5ZEYAKBMCI65Z6HP2VCFYUZ7E5RJOKYWXXA, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:13.728594026 (517h33m13.599677071s) [noise=true , address=45.91.116.6:4976 , piece_id=YAWNNDCLDBY6ZPEWGHTZK22SWYBDVL7IXULMJALJTOHOEBGOL45Q, seg_pos={0 0}, upload=454082]: Uploading piece... Dec 27 05:02:15.442938397 (517h33m15.314021432s) [upload=454082]: Next returned nil segment Dec 27 05:02:15.442957853 (517h33m15.314040899s) [upload=454082]: Waiting for error group managing segments... Dec 27 05:02:15.728204690 (517h33m15.599287726s) [noise=true , address=145.79.227.49:4337 , piece_id=F2R4QSOXQELSQRD6SJYCCI7ECN7GX5TPXIYKVCG6EQDWDLF3RAAQ, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.729084425 (517h33m15.60016747s) [noise=true , address=145.79.227.38:4978 , piece_id=BYOEMVXFDGNYWBTTDH2KND6G4OSXDMQ3GHZLFIOKSQQYZ5AHFITA, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.732214646 (517h33m15.603297681s) [noise=true , address=145.79.227.36:4518 , piece_id=UEX7HJ7JOGF36245ZLXJMCKCNT3HTAXTANA5N3YJ4FGNJVMGIJMQ, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.735941773 (517h33m15.607024818s) [noise=true , address=145.79.227.48:4302 , piece_id=MJEEFXBNQ2GDP6FC4354AUFMZ7ZNQRTA3UZKIXSPLYHRNPDB4YNA, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.736777655 (517h33m15.6078607s) [noise=true , address=145.79.227.11:4729 , piece_id=UJWE4UOAWWEJY43LP3I5R6XUB3E2AKQCK2KSAITJQPDRRXTP5A6A, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.751386853 (517h33m15.622469898s) [noise=true , address=145.79.227.52:4446 , piece_id=WVXCET4VDJEN5GXCYLHDGCWT7KQDUBJPRMTY4XDSHR3MINDIBQRQ, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.751453440 (517h33m15.622536475s) [noise=true , address=145.79.227.46:4602 , piece_id=W24GFKWDSLV2TWYIAX2G4MYZTXOGKVNVALM7FOYSKZ5MMZDLFEAQ, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.751533623 (517h33m15.622616668s) [noise=true , address=145.79.227.12:4765 , piece_id=CUEYAIJF2XLUZQU6Q33ZXNHQ7OOBSMKJ2ZLQMB6BIYALRAO6FG2Q, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.752806868 (517h33m15.623889913s) [noise=true , address=145.79.227.51:4409 , piece_id=K5TPWU3XHKQKFB5ZNQ5MRE7MGZEBQ5DGLFLIUJZS5UAHMMYBEIUA, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.772468056 (517h33m15.643551091s) [noise=true , address=145.79.227.37:4566 , piece_id=ECCVABXFVUH3LHMDR4KC35M3PZT7G3QKKXG7GGA6J5LXI2INFQEA, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.774243326 (517h33m15.645326371s) [noise=true , address=145.79.227.39:4013 , piece_id=RSRMT53QWCG2YYWNIK6ODEQ6LAIZDKXFDPIJELD4ETBIHALJSDZA, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:15.996738864 (517h33m15.867821909s) [noise=true , address=45.91.116.9:4262 , piece_id=DSJA5UGGUE322S5MOEOH27DNM4PPCB2BJ4F7CRALKOYJZWQBMGCQ, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:16.007683674 (517h33m15.878766719s) [noise=true , address=77.90.31.19:4841 , piece_id=VLPOQHQ2GFEJ7L7TG5ZEYAKBMCI65Z6HP2VCFYUZ7E5RJOKYWXXA, seg_pos={0 0}, upload=454082]: Done uploading piece. err: <nil> Dec 27 05:02:16.007709213 (517h33m15.878792248s) [seg_pos={0 0}, upload=454082]: Segment reached optimal threshold of 13 pieces. Dec 27 05:02:16.007987132 (517h33m15.879070187s) [noise=true , address=45.91.116.10:4082 , piece_id=QZQVUQO7YVHBI4ZNPGFCFSD3SUQUI2GHTQTGSMNYEJLY6T2HBZJA, seg_pos={0 0}, upload=454082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 27 05:02:16.007998333 (517h33m15.879081378s) [noise=true , address=45.91.116.10:4082 , piece_id=QZQVUQO7YVHBI4ZNPGFCFSD3SUQUI2GHTQTGSMNYEJLY6T2HBZJA, seg_pos={0 0}, upload=454082]: Long tail cancellation due to optimal threshold Dec 27 05:02:16.008167115 (517h33m15.87925015s) [noise=true , address=45.91.116.4:4937 , piece_id=T45QMUIHYFAH7LOHIBKF5P5UMTIHBYL3IX2R7L3BMZMMC5YDWOYQ, seg_pos={0 0}, upload=454082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 27 05:02:16.008173888 (517h33m15.879256923s) [noise=true , address=45.91.116.4:4937 , piece_id=T45QMUIHYFAH7LOHIBKF5P5UMTIHBYL3IX2R7L3BMZMMC5YDWOYQ, seg_pos={0 0}, upload=454082]: Long tail cancellation due to optimal threshold Dec 27 05:02:16.008240494 (517h33m15.87932357s) [noise=true , address=45.91.116.6:4976 , piece_id=YAWNNDCLDBY6ZPEWGHTZK22SWYBDVL7IXULMJALJTOHOEBGOL45Q, seg_pos={0 0}, upload=454082]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:42760->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:42760->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42760->45.91.116.6:4976: use of closed network connection Dec 27 05:02:16.008267045 (517h33m15.87935011s) [noise=true , address=45.91.116.6:4976 , piece_id=YAWNNDCLDBY6ZPEWGHTZK22SWYBDVL7IXULMJALJTOHOEBGOL45Q, seg_pos={0 0}, upload=454082]: Long tail cancellation due to optimal threshold Dec 27 05:02:16.008305849 (517h33m15.879388894s) [seg_pos={0 0}, upload=454082]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:02:16.008411851 (517h33m15.879494896s) [upload=454082]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:02:16.008418193 (517h33m15.879501228s) [upload=454082]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:02:16.300259997 (517h33m16.171343042s) [upload=454082]: Done uploading segments... Dec 27 05:02:16.300283272 (517h33m16.171366307s) [upload=454082]: Upload finished. err: <nil> Duration: 30.002788721s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 3.597031645s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 05:02:40 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.597065278s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X......................X...............] duration: 58.563396204s trace: https://jaeger.dev.storj.io/trace/60d70ba2fe18218 FAIL (2025-12-27 05:02:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 4 (517h19m18.685148379s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814531152 (517h19m18.685614187s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814852964 (517h19m18.685936009s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815170900 (517h19m18.686253945s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815486661 (517h19m18.686569706s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815835786 (517h19m18.686918831s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.816207564 (517h19m18.687290609s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818225116 (517h19m18.689308151s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818325547 (517h19m18.689408592s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.821132431 (517h19m18.692215466s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.822845684 (517h19m18.693928719s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.824299251 (517h19m18.695382296s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.825754102 (517h19m18.696837147s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.827122548 (517h19m18.698205593s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.828453653 (517h19m18.699536688s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:20.114191076 (517h19m19.985274121s) [upload=453889]: Next returned nil segment Dec 27 04:48:20.114206816 (517h19m19.985289851s) [upload=453889]: Waiting for error group managing segments... Dec 27 04:48:20.444209056 (517h19m20.315292102s) [noise=true , address=64.49.13.154:4929 , piece_id=2EXG7BEQ6NKAG3TNAH3TZ3K5ZI4KNOQYF3OEZMFWVX6BBMWNVXUQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.474597952 (517h19m20.345680987s) [noise=true , address=64.49.13.27:4964 , piece_id=XAROGTY2JLVFUVDYT7N7ZCABVYDMXA45NZF36NBAV4H72M3PJSCA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.478601025 (517h19m20.34968406s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.888413067 (517h19m20.759496102s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.930510167 (517h19m20.801593212s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.007760305 (517h19m20.87884334s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.041631362 (517h19m20.912714407s) [noise=true , address=77.90.31.14:4691 , piece_id=HIFLW7CWEYK6A6BZAYPDZLIUNA26IFXSFNVYV2UK4VPKEWYUMYJA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.052961556 (517h19m20.924044601s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:23.606703638 (517h19m23.477786673s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.046520345 (517h19m25.91760338s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.518771011 (517h19m26.389854046s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.589256778 (517h19m26.460339823s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:27.044280620 (517h19m26.915363655s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789125901 (517h19m36.660208936s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789145728 (517h19m36.660228764s) [seg_pos={0 0}, upload=453889]: Segment reached optimal threshold of 14 pieces. Dec 27 04:48:36.789615964 (517h19m36.660699009s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: read tcp 188.166.225.170:50246->145.79.227.40:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789624180 (517h19m36.660707215s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789761231 (517h19m36.660844276s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: read tcp 188.166.225.170:45976->145.79.227.44:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789766831 (517h19m36.660849866s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789924742 (517h19m36.661007788s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51): protocol: read tcp 188.166.225.170:43134->145.79.227.43:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789933699 (517h19m36.661016734s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790128380 (517h19m36.661211425s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1): protocol: read tcp 188.166.225.170:46462->145.79.227.25:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.790137838 (517h19m36.661220873s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790156633 (517h19m36.661239679s) [seg_pos={0 0}, upload=453889]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:48:36.790241626 (517h19m36.661324661s) [upload=453889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:48:36.790246826 (517h19m36.661329861s) [upload=453889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:48:37.084198870 (517h19m36.955281915s) [upload=453889]: Done uploading segments... Dec 27 04:48:37.084219929 (517h19m36.955302974s) [upload=453889]: Upload finished. err: <nil> Duration: 58.563470936s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 3.636989402s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 05:02:17 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.63700448s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [XXXX.X.X....X...XX.XXX..XXX..XXXX...XX...X.XX....XXXX..X.X.X] duration: 30.007147171s trace: https://jaeger.dev.storj.io/trace/103b79da2ab527ce FAIL (2025-12-27 05:01:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: VRI2YTUIULXQP7MCJZBMTW4YRNPGCQAV45ZX5A7MLMDE5EI2O4NA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: 7F4MYYS4QAO3WSOIJ4XXOVQONANEXBJXYOANXON6IA5XTSMJ3A6A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: LLFFXCRDOK7WJEVGADMGAFIYC7OXD2GXAXUZG5OEFDITISKQEG4A) context deadline exceeded ======LOGS====== d=454067]: Starting upload Dec 27 05:01:01.474967329 (517h32m1.346050374s) [upload=454067]: Done starting upload Dec 27 05:01:01.475047993 (517h32m1.346131038s) [upload=454067]: Uploading segments... Dec 27 05:01:01.475290113 (517h32m1.346373158s) [upload=454067]: Got next segment. Inline: false Dec 27 05:01:01.475364806 (517h32m1.346447841s) [upload=454067]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 05:01:01.475370788 (517h32m1.346453833s) [upload=454067]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 05:01:01.724199239 (517h32m1.595282354s) [seg_pos={0 0}, upload=454067]: Begin upload segment... Dec 27 05:01:01.724250506 (517h32m1.595333551s) [seg_pos={0 0}, upload=454067]: Done begin upload segment. Dec 27 05:01:01.724280273 (517h32m1.595363318s) [noise=true , address=145.79.227.37:4567 , piece_id=R5DVWT7OXLME3VJII34KMVXHULDJM2YUPDTOVXEQ56X4E4MJFWAA, seg_pos={0 0}, upload=454067]: Uploading piece... Dec 27 05:01:01.724395542 (517h32m1.595478577s) [noise=true , address=145.79.227.50:4374 , piece_id=34RRKX3CLP5JB5SYKID7OASZP2CDC2SAVNXNAXRVS4BNSTDOZ4GQ, seg_pos={0 0}, upload=454067]: Uploading piece... Dec 27 05:01:01.725533018 (517h32m1.596616063s) [noise=true , address=145.79.227.11:4732 , piece_id=A3YYQGPUTTJUHJKPNBND34FEMO5RK5WGWLSSU6V62A2JACKRYKQQ, seg_pos={0 0}, upload=454067]: Uploading piece... Dec 27 05:01:01.726307803 (517h32m1.597390889s) [noise=true , address=45.91.116.9:4263 , piece_id=VRI2YTUIULXQP7MCJZBMTW4YRNPGCQAV45ZX5A7MLMDE5EI2O4NA, seg_pos={0 0}, upload=454067]: Uploading piece... Dec 27 05:01:01.727332645 (517h32m1.598415691s) [noise=true , address=145.79.227.16:4912 , piece_id=Z4WWSODLVICMP5INR27QB4TYPM674CW7JECDQTLN2AIWFHKHABCQ, seg_pos={0 0}, upload=454067]: Uploading piece... Dec 27 05:01:01.728855484 (517h32m1.59993853s) [noise=true , address=145.79.227.20:4439 , piece_id=XAAJSVTGELLDZPGG2XHXDOL6KVORTVEMGPZPTHQMXD3UWZOVFMJA, seg_pos={0 0}, upload=454067]: Uploading piece... Dec 27 05:01:01.729683882 (517h32m1.600766927s) [noise=true , address=77.90.31.15:4697 , piece_id=MZB3UQI62FX4ZG6PXD7DR2LCNJUIB77EI4MICXZBCYGNF5UE3JIA, seg_pos={0 0}, upload=454067]: Uploading piece... Dec 27 05:01:01.730303562 (517h32m1.601386607s) [noise=true , address=77.90.31.21:4878 , piece_id=TQOHC3DIXY645RKQFQTGGTEN4OS2CPMW65VUTI4R65ONOQM4NFCQ, seg_pos={0 0}, upload=454067]: Uploading piece... Dec 27 05:01:01.732024088 (517h32m1.603107123s) [noise=true , address=45.91.116.6:4973 , piece_id=7F4MYYS4QAO3WSOIJ4XXOVQONANEXBJXYOANXON6IA5XTSMJ3A6A, seg_pos={0 0}, upload=454067]: Uploading piece... Dec 27 05:01:01.733021137 (517h32m1.604104192s) [noise=true , address=45.91.116.14:4190 , piece_id=LLFFXCRDOK7WJEVGADMGAFIYC7OXD2GXAXUZG5OEFDITISKQEG4A, seg_pos={0 0}, upload=454067]: Uploading piece... Dec 27 05:01:01.733469290 (517h32m1.604552336s) [noise=true , address=77.90.31.14:4662 , piece_id=SNUBXFFP3STWSJ2FWH3ZBFSHUK3DGK6LKHWXTH2YJB5S2TPY46LQ, seg_pos={0 0}, upload=454067]: Uploading piece... Dec 27 05:01:01.735035342 (517h32m1.606118388s) [noise=true , address=145.79.227.47:4267 , piece_id=2WGZL34MN4SZVXDCT2ENDHDFH3VQRFWWQ2YIY6GJQEVFHC4LX6XA, seg_pos={0 0}, upload=454067]: Uploading piece... Dec 27 05:01:01.738257307 (517h32m1.609340352s) [noise=true , address=145.79.227.4:4185 , piece_id=QJI7OIGX4MC6BA3LWNUOGDVT2SEFIIRGCBPX6MVXUOSZI3IV2N6A, seg_pos={0 0}, upload=454067]: Uploading piece... Dec 27 05:01:01.743337471 (517h32m1.614420516s) [noise=true , address=145.79.227.36:4519 , piece_id=ZJIPIB6HKTWZ575QWTT6YHHGO3HR7JVSKM3WXOFXEGK3C5G4H6IQ, seg_pos={0 0}, upload=454067]: Uploading piece... Dec 27 05:01:01.746220681 (517h32m1.617303726s) [noise=true , address=145.79.227.49:4339 , piece_id=IP2VLJ7Y5EY3PMVV37RACZORSMJZ6VEOO4D3LGLNYCIBB63DI4HA, seg_pos={0 0}, upload=454067]: Uploading piece... Dec 27 05:01:01.746581898 (517h32m1.617664943s) [noise=true , address=145.79.227.48:4303 , piece_id=VG4Q54QWJBUSMTBQR7FWOB5QUQJBPQCQRT3IWJPUA3XGTTAEWZQA, seg_pos={0 0}, upload=454067]: Uploading piece... Dec 27 05:01:03.353048594 (517h32m3.224131639s) [upload=454067]: Next returned nil segment Dec 27 05:01:03.353071037 (517h32m3.224154072s) [upload=454067]: Waiting for error group managing segments... Dec 27 05:01:03.737210592 (517h32m3.608293627s) [noise=true , address=145.79.227.11:4732 , piece_id=A3YYQGPUTTJUHJKPNBND34FEMO5RK5WGWLSSU6V62A2JACKRYKQQ, seg_pos={0 0}, upload=454067]: Done uploading piece. err: <nil> Dec 27 05:01:03.741422312 (517h32m3.612505357s) [noise=true , address=145.79.227.50:4374 , piece_id=34RRKX3CLP5JB5SYKID7OASZP2CDC2SAVNXNAXRVS4BNSTDOZ4GQ, seg_pos={0 0}, upload=454067]: Done uploading piece. err: <nil> Dec 27 05:01:03.746336258 (517h32m3.617419304s) [noise=true , address=145.79.227.16:4912 , piece_id=Z4WWSODLVICMP5INR27QB4TYPM674CW7JECDQTLN2AIWFHKHABCQ, seg_pos={0 0}, upload=454067]: Done uploading piece. err: <nil> Dec 27 05:01:03.748681955 (517h32m3.619765s) [noise=true , address=145.79.227.49:4339 , piece_id=IP2VLJ7Y5EY3PMVV37RACZORSMJZ6VEOO4D3LGLNYCIBB63DI4HA, seg_pos={0 0}, upload=454067]: Done uploading piece. err: <nil> Dec 27 05:01:03.755416259 (517h32m3.626499334s) [noise=true , address=145.79.227.20:4439 , piece_id=XAAJSVTGELLDZPGG2XHXDOL6KVORTVEMGPZPTHQMXD3UWZOVFMJA, seg_pos={0 0}, upload=454067]: Done uploading piece. err: <nil> Dec 27 05:01:03.764720525 (517h32m3.63580358s) [noise=true , address=145.79.227.48:4303 , piece_id=VG4Q54QWJBUSMTBQR7FWOB5QUQJBPQCQRT3IWJPUA3XGTTAEWZQA, seg_pos={0 0}, upload=454067]: Done uploading piece. err: <nil> Dec 27 05:01:03.767858400 (517h32m3.638941445s) [noise=true , address=145.79.227.37:4567 , piece_id=R5DVWT7OXLME3VJII34KMVXHULDJM2YUPDTOVXEQ56X4E4MJFWAA, seg_pos={0 0}, upload=454067]: Done uploading piece. err: <nil> Dec 27 05:01:03.769089504 (517h32m3.640172539s) [noise=true , address=145.79.227.4:4185 , piece_id=QJI7OIGX4MC6BA3LWNUOGDVT2SEFIIRGCBPX6MVXUOSZI3IV2N6A, seg_pos={0 0}, upload=454067]: Done uploading piece. err: <nil> Dec 27 05:01:03.786204594 (517h32m3.657287629s) [noise=true , address=145.79.227.36:4519 , piece_id=ZJIPIB6HKTWZ575QWTT6YHHGO3HR7JVSKM3WXOFXEGK3C5G4H6IQ, seg_pos={0 0}, upload=454067]: Done uploading piece. err: <nil> Dec 27 05:01:03.787706945 (517h32m3.65878999s) [noise=true , address=145.79.227.47:4267 , piece_id=2WGZL34MN4SZVXDCT2ENDHDFH3VQRFWWQ2YIY6GJQEVFHC4LX6XA, seg_pos={0 0}, upload=454067]: Done uploading piece. err: <nil> Dec 27 05:01:03.976021427 (517h32m3.847104472s) [noise=true , address=45.91.116.9:4263 , piece_id=VRI2YTUIULXQP7MCJZBMTW4YRNPGCQAV45ZX5A7MLMDE5EI2O4NA, seg_pos={0 0}, upload=454067]: Done uploading piece. err: <nil> Dec 27 05:01:04.011251140 (517h32m3.882334185s) [noise=true , address=45.91.116.14:4190 , piece_id=LLFFXCRDOK7WJEVGADMGAFIYC7OXD2GXAXUZG5OEFDITISKQEG4A, seg_pos={0 0}, upload=454067]: Done uploading piece. err: <nil> Dec 27 05:01:04.133642332 (517h32m4.004725367s) [noise=true , address=45.91.116.6:4973 , piece_id=7F4MYYS4QAO3WSOIJ4XXOVQONANEXBJXYOANXON6IA5XTSMJ3A6A, seg_pos={0 0}, upload=454067]: Done uploading piece. err: <nil> Dec 27 05:01:04.133664635 (517h32m4.00474768s) [seg_pos={0 0}, upload=454067]: Segment reached optimal threshold of 13 pieces. Dec 27 05:01:04.133927174 (517h32m4.005010219s) [noise=true , address=77.90.31.14:4662 , piece_id=SNUBXFFP3STWSJ2FWH3ZBFSHUK3DGK6LKHWXTH2YJB5S2TPY46LQ, seg_pos={0 0}, upload=454067]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:49140->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:49140->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49140->77.90.31.14:4662: use of closed network connection Dec 27 05:01:04.133937134 (517h32m4.005020179s) [noise=true , address=77.90.31.14:4662 , piece_id=SNUBXFFP3STWSJ2FWH3ZBFSHUK3DGK6LKHWXTH2YJB5S2TPY46LQ, seg_pos={0 0}, upload=454067]: Long tail cancellation due to optimal threshold Dec 27 05:01:04.134129680 (517h32m4.005212715s) [noise=true , address=77.90.31.15:4697 , piece_id=MZB3UQI62FX4ZG6PXD7DR2LCNJUIB77EI4MICXZBCYGNF5UE3JIA, seg_pos={0 0}, upload=454067]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 27 05:01:04.134138016 (517h32m4.005221061s) [noise=true , address=77.90.31.15:4697 , piece_id=MZB3UQI62FX4ZG6PXD7DR2LCNJUIB77EI4MICXZBCYGNF5UE3JIA, seg_pos={0 0}, upload=454067]: Long tail cancellation due to optimal threshold Dec 27 05:01:04.134863848 (517h32m4.005946893s) [noise=true , address=77.90.31.21:4878 , piece_id=TQOHC3DIXY645RKQFQTGGTEN4OS2CPMW65VUTI4R65ONOQM4NFCQ, seg_pos={0 0}, upload=454067]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: read tcp 188.166.225.170:40426->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 05:01:04.134886672 (517h32m4.005969717s) [noise=true , address=77.90.31.21:4878 , piece_id=TQOHC3DIXY645RKQFQTGGTEN4OS2CPMW65VUTI4R65ONOQM4NFCQ, seg_pos={0 0}, upload=454067]: Long tail cancellation due to optimal threshold Dec 27 05:01:04.134963478 (517h32m4.006046523s) [seg_pos={0 0}, upload=454067]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 05:01:04.135305129 (517h32m4.006388184s) [upload=454067]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 05:01:04.135315077 (517h32m4.006398122s) [upload=454067]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 05:01:04.470828489 (517h32m4.341911535s) [upload=454067]: Done uploading segments... Dec 27 05:01:04.470847546 (517h32m4.341930581s) [upload=454067]: Upload finished. err: <nil> Duration: 30.007173742s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X......................X..............] duration: 34.163236195s trace: https://jaeger.dev.storj.io/trace/60d70ba2fe18218 FAIL (2025-12-27 05:01:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 4 (517h19m18.685148379s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814531152 (517h19m18.685614187s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814852964 (517h19m18.685936009s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815170900 (517h19m18.686253945s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815486661 (517h19m18.686569706s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815835786 (517h19m18.686918831s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.816207564 (517h19m18.687290609s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818225116 (517h19m18.689308151s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818325547 (517h19m18.689408592s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.821132431 (517h19m18.692215466s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.822845684 (517h19m18.693928719s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.824299251 (517h19m18.695382296s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.825754102 (517h19m18.696837147s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.827122548 (517h19m18.698205593s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.828453653 (517h19m18.699536688s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:20.114191076 (517h19m19.985274121s) [upload=453889]: Next returned nil segment Dec 27 04:48:20.114206816 (517h19m19.985289851s) [upload=453889]: Waiting for error group managing segments... Dec 27 04:48:20.444209056 (517h19m20.315292102s) [noise=true , address=64.49.13.154:4929 , piece_id=2EXG7BEQ6NKAG3TNAH3TZ3K5ZI4KNOQYF3OEZMFWVX6BBMWNVXUQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.474597952 (517h19m20.345680987s) [noise=true , address=64.49.13.27:4964 , piece_id=XAROGTY2JLVFUVDYT7N7ZCABVYDMXA45NZF36NBAV4H72M3PJSCA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.478601025 (517h19m20.34968406s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.888413067 (517h19m20.759496102s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.930510167 (517h19m20.801593212s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.007760305 (517h19m20.87884334s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.041631362 (517h19m20.912714407s) [noise=true , address=77.90.31.14:4691 , piece_id=HIFLW7CWEYK6A6BZAYPDZLIUNA26IFXSFNVYV2UK4VPKEWYUMYJA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.052961556 (517h19m20.924044601s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:23.606703638 (517h19m23.477786673s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.046520345 (517h19m25.91760338s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.518771011 (517h19m26.389854046s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.589256778 (517h19m26.460339823s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:27.044280620 (517h19m26.915363655s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789125901 (517h19m36.660208936s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789145728 (517h19m36.660228764s) [seg_pos={0 0}, upload=453889]: Segment reached optimal threshold of 14 pieces. Dec 27 04:48:36.789615964 (517h19m36.660699009s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: read tcp 188.166.225.170:50246->145.79.227.40:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789624180 (517h19m36.660707215s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789761231 (517h19m36.660844276s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: read tcp 188.166.225.170:45976->145.79.227.44:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789766831 (517h19m36.660849866s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789924742 (517h19m36.661007788s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51): protocol: read tcp 188.166.225.170:43134->145.79.227.43:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789933699 (517h19m36.661016734s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790128380 (517h19m36.661211425s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1): protocol: read tcp 188.166.225.170:46462->145.79.227.25:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.790137838 (517h19m36.661220873s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790156633 (517h19m36.661239679s) [seg_pos={0 0}, upload=453889]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:48:36.790241626 (517h19m36.661324661s) [upload=453889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:48:36.790246826 (517h19m36.661329861s) [upload=453889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:48:37.084198870 (517h19m36.955281915s) [upload=453889]: Done uploading segments... Dec 27 04:48:37.084219929 (517h19m36.955302974s) [upload=453889]: Upload finished. err: <nil> Duration: 34.163250614s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 3.563106898s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 05:01:04 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.563162846s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X......................X.............] duration: 53.87083472s trace: https://jaeger.dev.storj.io/trace/60d70ba2fe18218 FAIL (2025-12-27 05:00:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 4 (517h19m18.685148379s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814531152 (517h19m18.685614187s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814852964 (517h19m18.685936009s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815170900 (517h19m18.686253945s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815486661 (517h19m18.686569706s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815835786 (517h19m18.686918831s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.816207564 (517h19m18.687290609s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818225116 (517h19m18.689308151s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818325547 (517h19m18.689408592s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.821132431 (517h19m18.692215466s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.822845684 (517h19m18.693928719s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.824299251 (517h19m18.695382296s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.825754102 (517h19m18.696837147s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.827122548 (517h19m18.698205593s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.828453653 (517h19m18.699536688s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:20.114191076 (517h19m19.985274121s) [upload=453889]: Next returned nil segment Dec 27 04:48:20.114206816 (517h19m19.985289851s) [upload=453889]: Waiting for error group managing segments... Dec 27 04:48:20.444209056 (517h19m20.315292102s) [noise=true , address=64.49.13.154:4929 , piece_id=2EXG7BEQ6NKAG3TNAH3TZ3K5ZI4KNOQYF3OEZMFWVX6BBMWNVXUQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.474597952 (517h19m20.345680987s) [noise=true , address=64.49.13.27:4964 , piece_id=XAROGTY2JLVFUVDYT7N7ZCABVYDMXA45NZF36NBAV4H72M3PJSCA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.478601025 (517h19m20.34968406s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.888413067 (517h19m20.759496102s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.930510167 (517h19m20.801593212s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.007760305 (517h19m20.87884334s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.041631362 (517h19m20.912714407s) [noise=true , address=77.90.31.14:4691 , piece_id=HIFLW7CWEYK6A6BZAYPDZLIUNA26IFXSFNVYV2UK4VPKEWYUMYJA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.052961556 (517h19m20.924044601s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:23.606703638 (517h19m23.477786673s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.046520345 (517h19m25.91760338s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.518771011 (517h19m26.389854046s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.589256778 (517h19m26.460339823s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:27.044280620 (517h19m26.915363655s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789125901 (517h19m36.660208936s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789145728 (517h19m36.660228764s) [seg_pos={0 0}, upload=453889]: Segment reached optimal threshold of 14 pieces. Dec 27 04:48:36.789615964 (517h19m36.660699009s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: read tcp 188.166.225.170:50246->145.79.227.40:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789624180 (517h19m36.660707215s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789761231 (517h19m36.660844276s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: read tcp 188.166.225.170:45976->145.79.227.44:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789766831 (517h19m36.660849866s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789924742 (517h19m36.661007788s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51): protocol: read tcp 188.166.225.170:43134->145.79.227.43:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789933699 (517h19m36.661016734s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790128380 (517h19m36.661211425s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1): protocol: read tcp 188.166.225.170:46462->145.79.227.25:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.790137838 (517h19m36.661220873s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790156633 (517h19m36.661239679s) [seg_pos={0 0}, upload=453889]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:48:36.790241626 (517h19m36.661324661s) [upload=453889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:48:36.790246826 (517h19m36.661329861s) [upload=453889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:48:37.084198870 (517h19m36.955281915s) [upload=453889]: Done uploading segments... Dec 27 04:48:37.084219929 (517h19m36.955302974s) [upload=453889]: Upload finished. err: <nil> Duration: 53.870849407s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.XXXX.X.X....X...XX.XXX..XXX..XXXX...XX...X.XX....XXXX..X.X.] duration: 6.027411767s trace: https://jaeger.dev.storj.io/trace/c2caf99ff0bd8e3 FAIL (2025-12-27 05:00:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: 2RAUNHDOPUIWT57I3VFZTFLTDFQG2QL6WX5YKFQKFGENSBGKJJBA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: PNSLBIFTV7OENCOY2VRBFK3XPV2SWITSRUIPFBZ7LLO5KZQVLJYQ) context deadline exceeded ======LOGS====== Dec 27 04:59:46.153172985 (517h30m46.02425605s) [upload=454047]: Starting upload Dec 27 04:59:46.153295929 (517h30m46.024378984s) [upload=454047]: Done starting upload Dec 27 04:59:46.153402713 (517h30m46.024485748s) [upload=454047]: Uploading segments... Dec 27 04:59:46.153674230 (517h30m46.024757275s) [upload=454047]: Got next segment. Inline: false Dec 27 04:59:46.153720047 (517h30m46.024803102s) [upload=454047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:59:46.153725958 (517h30m46.024809013s) [upload=454047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:59:46.399300094 (517h30m46.270383129s) [seg_pos={0 0}, upload=454047]: Begin upload segment... Dec 27 04:59:46.399359627 (517h30m46.270442663s) [seg_pos={0 0}, upload=454047]: Done begin upload segment. Dec 27 04:59:46.399416886 (517h30m46.270499921s) [noise=true , address=145.79.227.48:4302 , piece_id=T2QFAB5QKNYUCPQ6N5BY7E2CL3TNYE7VZUSIOLAPAKRQCW6WA2GA, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.399515384 (517h30m46.270598419s) [noise=true , address=45.91.116.5:4299 , piece_id=FWUY6XRZ34UZY44Q4Q3TEWRH5PS5WATSSLMYC2VI6LPEFV6XJZCA, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.400030284 (517h30m46.271113329s) [noise=true , address=77.90.31.16:4735 , piece_id=EMVLUIFA4LAIC74HUQLQELPIYJVJ7NALJPO7TP7GODGQ3LNFTFIQ, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.400328723 (517h30m46.271411758s) [noise=true , address=77.90.31.11:4556 , piece_id=6AGQQJEOZD5CYRDHSCG6IGIZBDKFEGEHTFJ2TGJFQ3ODECT367EQ, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.400614417 (517h30m46.271697452s) [noise=true , address=45.91.116.12:4227 , piece_id=2RAUNHDOPUIWT57I3VFZTFLTDFQG2QL6WX5YKFQKFGENSBGKJJBA, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.400917755 (517h30m46.2720008s) [noise=true , address=45.91.116.13:4153 , piece_id=HJ5XOXJFEI5JNLYK4TSZFZBZZAWR7NBVTA2EVMKSECEOCQA377FQ, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.401243925 (517h30m46.27232696s) [noise=true , address=145.79.227.28:4725 , piece_id=U3WTJJMPYMYJGOAUOOTVFCSSQHSPQWZ7CLCF4O5I5VKJI2MWSNMQ, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.401571930 (517h30m46.272654965s) [noise=true , address=145.79.227.6:4259 , piece_id=3MDQTKVA7EQGRRKQBXNTSW4ESHWM6343HDGYE2MOIK2Q3A3DJZUQ, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.401966131 (517h30m46.273049176s) [noise=true , address=145.79.227.50:4373 , piece_id=BAY7LKWLSSO3KW3JEKTXO3AHFEKPOXIEZMOKDVM42CKR76Z76ATA, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.402181230 (517h30m46.273264275s) [noise=true , address=145.79.227.46:4601 , piece_id=M6LZ3Q6WDWFK2E3ASY5TJ6UO5AOY3K5E4JG72OJJCHSWTN5MK2XA, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.402297040 (517h30m46.273380085s) [noise=true , address=145.79.227.37:4565 , piece_id=NQRYLDU72RS7LPYKSL34EDJJBQENXU72QCOBU3TZZCRW22KCXDCQ, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.402705138 (517h30m46.273788183s) [noise=true , address=145.79.227.47:4266 , piece_id=A3AO6DOOC7RO4CZWA3OCXWHJZEWYBOSWZN42AHTYLUBW26HRGPOA, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.402730867 (517h30m46.273813912s) [noise=true , address=145.79.227.3:4151 , piece_id=T5BZO7VAI5IQQA3W4PIEM6VCYS52KI44WB7ZE7KRTJXORZ5V5ACA, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.403098036 (517h30m46.274181071s) [noise=true , address=145.79.227.42:4121 , piece_id=JYA7LUS7PDUML2XQ3TKFG7LVBSQS6U2TOUZQZDJHBONCQK2OQ7JA, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.403464935 (517h30m46.27454798s) [noise=true , address=145.79.227.23:4546 , piece_id=SDALLDC7VDZIBLXEVZYEAFMSNR3BHN6U3M3B6XHWYJMAE2GODXEA, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.403785986 (517h30m46.274869021s) [noise=true , address=77.90.31.15:4700 , piece_id=PNSLBIFTV7OENCOY2VRBFK3XPV2SWITSRUIPFBZ7LLO5KZQVLJYQ, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:48.122734317 (517h30m47.993817352s) [upload=454047]: Next returned nil segment Dec 27 04:59:48.122747102 (517h30m47.993830137s) [upload=454047]: Waiting for error group managing segments... Dec 27 04:59:48.403461540 (517h30m48.274544575s) [noise=true , address=145.79.227.50:4373 , piece_id=BAY7LKWLSSO3KW3JEKTXO3AHFEKPOXIEZMOKDVM42CKR76Z76ATA, seg_pos={0 0}, upload=454047]: Done uploading piece. err: <nil> Dec 27 04:59:48.406771843 (517h30m48.277854878s) [noise=true , address=145.79.227.46:4601 , piece_id=M6LZ3Q6WDWFK2E3ASY5TJ6UO5AOY3K5E4JG72OJJCHSWTN5MK2XA, seg_pos={0 0}, upload=454047]: Done uploading piece. err: <nil> Dec 27 04:59:48.407361226 (517h30m48.278444261s) [noise=true , address=145.79.227.23:4546 , piece_id=SDALLDC7VDZIBLXEVZYEAFMSNR3BHN6U3M3B6XHWYJMAE2GODXEA, seg_pos={0 0}, upload=454047]: Done uploading piece. err: <nil> Dec 27 04:59:48.408696699 (517h30m48.279779734s) [noise=true , address=145.79.227.48:4302 , piece_id=T2QFAB5QKNYUCPQ6N5BY7E2CL3TNYE7VZUSIOLAPAKRQCW6WA2GA, seg_pos={0 0}, upload=454047]: Done uploading piece. err: <nil> Dec 27 04:59:48.417150977 (517h30m48.288234012s) [noise=true , address=145.79.227.47:4266 , piece_id=A3AO6DOOC7RO4CZWA3OCXWHJZEWYBOSWZN42AHTYLUBW26HRGPOA, seg_pos={0 0}, upload=454047]: Done uploading piece. err: <nil> Dec 27 04:59:48.424662841 (517h30m48.295745886s) [noise=true , address=145.79.227.28:4725 , piece_id=U3WTJJMPYMYJGOAUOOTVFCSSQHSPQWZ7CLCF4O5I5VKJI2MWSNMQ, seg_pos={0 0}, upload=454047]: Done uploading piece. err: <nil> Dec 27 04:59:48.426624607 (517h30m48.297707642s) [noise=true , address=145.79.227.6:4259 , piece_id=3MDQTKVA7EQGRRKQBXNTSW4ESHWM6343HDGYE2MOIK2Q3A3DJZUQ, seg_pos={0 0}, upload=454047]: Done uploading piece. err: <nil> Dec 27 04:59:48.428890933 (517h30m48.299973968s) [noise=true , address=145.79.227.42:4121 , piece_id=JYA7LUS7PDUML2XQ3TKFG7LVBSQS6U2TOUZQZDJHBONCQK2OQ7JA, seg_pos={0 0}, upload=454047]: Done uploading piece. err: <nil> Dec 27 04:59:48.432133296 (517h30m48.303216331s) [noise=true , address=145.79.227.37:4565 , piece_id=NQRYLDU72RS7LPYKSL34EDJJBQENXU72QCOBU3TZZCRW22KCXDCQ, seg_pos={0 0}, upload=454047]: Done uploading piece. err: <nil> Dec 27 04:59:48.435291590 (517h30m48.306374635s) [noise=true , address=145.79.227.3:4151 , piece_id=T5BZO7VAI5IQQA3W4PIEM6VCYS52KI44WB7ZE7KRTJXORZ5V5ACA, seg_pos={0 0}, upload=454047]: Done uploading piece. err: <nil> Dec 27 04:59:48.608504507 (517h30m48.479587542s) [noise=true , address=77.90.31.15:4700 , piece_id=PNSLBIFTV7OENCOY2VRBFK3XPV2SWITSRUIPFBZ7LLO5KZQVLJYQ, seg_pos={0 0}, upload=454047]: Done uploading piece. err: <nil> Dec 27 04:59:48.624416867 (517h30m48.495499902s) [noise=true , address=45.91.116.5:4299 , piece_id=FWUY6XRZ34UZY44Q4Q3TEWRH5PS5WATSSLMYC2VI6LPEFV6XJZCA, seg_pos={0 0}, upload=454047]: Done uploading piece. err: <nil> Dec 27 04:59:48.659913960 (517h30m48.530996995s) [noise=true , address=45.91.116.12:4227 , piece_id=2RAUNHDOPUIWT57I3VFZTFLTDFQG2QL6WX5YKFQKFGENSBGKJJBA, seg_pos={0 0}, upload=454047]: Done uploading piece. err: <nil> Dec 27 04:59:48.659933306 (517h30m48.531016341s) [seg_pos={0 0}, upload=454047]: Segment reached optimal threshold of 13 pieces. Dec 27 04:59:48.660109081 (517h30m48.531192116s) [noise=true , address=77.90.31.11:4556 , piece_id=6AGQQJEOZD5CYRDHSCG6IGIZBDKFEGEHTFJ2TGJFQ3ODECT367EQ, seg_pos={0 0}, upload=454047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 27 04:59:48.660115173 (517h30m48.531198218s) [noise=true , address=77.90.31.11:4556 , piece_id=6AGQQJEOZD5CYRDHSCG6IGIZBDKFEGEHTFJ2TGJFQ3ODECT367EQ, seg_pos={0 0}, upload=454047]: Long tail cancellation due to optimal threshold Dec 27 04:59:48.660318710 (517h30m48.531401755s) [noise=true , address=77.90.31.16:4735 , piece_id=EMVLUIFA4LAIC74HUQLQELPIYJVJ7NALJPO7TP7GODGQ3LNFTFIQ, seg_pos={0 0}, upload=454047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 27 04:59:48.660336524 (517h30m48.531419559s) [noise=true , address=77.90.31.16:4735 , piece_id=EMVLUIFA4LAIC74HUQLQELPIYJVJ7NALJPO7TP7GODGQ3LNFTFIQ, seg_pos={0 0}, upload=454047]: Long tail cancellation due to optimal threshold Dec 27 04:59:48.660426535 (517h30m48.53150957s) [noise=true , address=45.91.116.13:4153 , piece_id=HJ5XOXJFEI5JNLYK4TSZFZBZZAWR7NBVTA2EVMKSECEOCQA377FQ, seg_pos={0 0}, upload=454047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 27 04:59:48.660432257 (517h30m48.531515302s) [noise=true , address=45.91.116.13:4153 , piece_id=HJ5XOXJFEI5JNLYK4TSZFZBZZAWR7NBVTA2EVMKSECEOCQA377FQ, seg_pos={0 0}, upload=454047]: Long tail cancellation due to optimal threshold Dec 27 04:59:48.660448948 (517h30m48.531531983s) [seg_pos={0 0}, upload=454047]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:59:48.660536856 (517h30m48.531619891s) [upload=454047]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:59:48.660543709 (517h30m48.531626744s) [upload=454047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:59:48.950791860 (517h30m48.821874905s) [upload=454047]: Done uploading segments... Dec 27 04:59:48.950810295 (517h30m48.82189333s) [upload=454047]: Upload finished. err: <nil> Duration: 6.027443929s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [..XXXX.X.X....X...XX.XXX..XXX..XXXX...XX...X.XX....XXXX..X.X] duration: 30.003215972s trace: https://jaeger.dev.storj.io/trace/c2caf99ff0bd8e3 FAIL (2025-12-27 05:00:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: 2RAUNHDOPUIWT57I3VFZTFLTDFQG2QL6WX5YKFQKFGENSBGKJJBA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: PNSLBIFTV7OENCOY2VRBFK3XPV2SWITSRUIPFBZ7LLO5KZQVLJYQ) context deadline exceeded ======LOGS====== Dec 27 04:59:46.153172985 (517h30m46.02425605s) [upload=454047]: Starting upload Dec 27 04:59:46.153295929 (517h30m46.024378984s) [upload=454047]: Done starting upload Dec 27 04:59:46.153402713 (517h30m46.024485748s) [upload=454047]: Uploading segments... Dec 27 04:59:46.153674230 (517h30m46.024757275s) [upload=454047]: Got next segment. Inline: false Dec 27 04:59:46.153720047 (517h30m46.024803102s) [upload=454047]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:59:46.153725958 (517h30m46.024809013s) [upload=454047]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:59:46.399300094 (517h30m46.270383129s) [seg_pos={0 0}, upload=454047]: Begin upload segment... Dec 27 04:59:46.399359627 (517h30m46.270442663s) [seg_pos={0 0}, upload=454047]: Done begin upload segment. Dec 27 04:59:46.399416886 (517h30m46.270499921s) [noise=true , address=145.79.227.48:4302 , piece_id=T2QFAB5QKNYUCPQ6N5BY7E2CL3TNYE7VZUSIOLAPAKRQCW6WA2GA, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.399515384 (517h30m46.270598419s) [noise=true , address=45.91.116.5:4299 , piece_id=FWUY6XRZ34UZY44Q4Q3TEWRH5PS5WATSSLMYC2VI6LPEFV6XJZCA, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.400030284 (517h30m46.271113329s) [noise=true , address=77.90.31.16:4735 , piece_id=EMVLUIFA4LAIC74HUQLQELPIYJVJ7NALJPO7TP7GODGQ3LNFTFIQ, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.400328723 (517h30m46.271411758s) [noise=true , address=77.90.31.11:4556 , piece_id=6AGQQJEOZD5CYRDHSCG6IGIZBDKFEGEHTFJ2TGJFQ3ODECT367EQ, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.400614417 (517h30m46.271697452s) [noise=true , address=45.91.116.12:4227 , piece_id=2RAUNHDOPUIWT57I3VFZTFLTDFQG2QL6WX5YKFQKFGENSBGKJJBA, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.400917755 (517h30m46.2720008s) [noise=true , address=45.91.116.13:4153 , piece_id=HJ5XOXJFEI5JNLYK4TSZFZBZZAWR7NBVTA2EVMKSECEOCQA377FQ, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.401243925 (517h30m46.27232696s) [noise=true , address=145.79.227.28:4725 , piece_id=U3WTJJMPYMYJGOAUOOTVFCSSQHSPQWZ7CLCF4O5I5VKJI2MWSNMQ, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.401571930 (517h30m46.272654965s) [noise=true , address=145.79.227.6:4259 , piece_id=3MDQTKVA7EQGRRKQBXNTSW4ESHWM6343HDGYE2MOIK2Q3A3DJZUQ, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.401966131 (517h30m46.273049176s) [noise=true , address=145.79.227.50:4373 , piece_id=BAY7LKWLSSO3KW3JEKTXO3AHFEKPOXIEZMOKDVM42CKR76Z76ATA, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.402181230 (517h30m46.273264275s) [noise=true , address=145.79.227.46:4601 , piece_id=M6LZ3Q6WDWFK2E3ASY5TJ6UO5AOY3K5E4JG72OJJCHSWTN5MK2XA, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.402297040 (517h30m46.273380085s) [noise=true , address=145.79.227.37:4565 , piece_id=NQRYLDU72RS7LPYKSL34EDJJBQENXU72QCOBU3TZZCRW22KCXDCQ, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.402705138 (517h30m46.273788183s) [noise=true , address=145.79.227.47:4266 , piece_id=A3AO6DOOC7RO4CZWA3OCXWHJZEWYBOSWZN42AHTYLUBW26HRGPOA, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.402730867 (517h30m46.273813912s) [noise=true , address=145.79.227.3:4151 , piece_id=T5BZO7VAI5IQQA3W4PIEM6VCYS52KI44WB7ZE7KRTJXORZ5V5ACA, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.403098036 (517h30m46.274181071s) [noise=true , address=145.79.227.42:4121 , piece_id=JYA7LUS7PDUML2XQ3TKFG7LVBSQS6U2TOUZQZDJHBONCQK2OQ7JA, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.403464935 (517h30m46.27454798s) [noise=true , address=145.79.227.23:4546 , piece_id=SDALLDC7VDZIBLXEVZYEAFMSNR3BHN6U3M3B6XHWYJMAE2GODXEA, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:46.403785986 (517h30m46.274869021s) [noise=true , address=77.90.31.15:4700 , piece_id=PNSLBIFTV7OENCOY2VRBFK3XPV2SWITSRUIPFBZ7LLO5KZQVLJYQ, seg_pos={0 0}, upload=454047]: Uploading piece... Dec 27 04:59:48.122734317 (517h30m47.993817352s) [upload=454047]: Next returned nil segment Dec 27 04:59:48.122747102 (517h30m47.993830137s) [upload=454047]: Waiting for error group managing segments... Dec 27 04:59:48.403461540 (517h30m48.274544575s) [noise=true , address=145.79.227.50:4373 , piece_id=BAY7LKWLSSO3KW3JEKTXO3AHFEKPOXIEZMOKDVM42CKR76Z76ATA, seg_pos={0 0}, upload=454047]: Done uploading piece. err: <nil> Dec 27 04:59:48.406771843 (517h30m48.277854878s) [noise=true , address=145.79.227.46:4601 , piece_id=M6LZ3Q6WDWFK2E3ASY5TJ6UO5AOY3K5E4JG72OJJCHSWTN5MK2XA, seg_pos={0 0}, upload=454047]: Done uploading piece. err: <nil> Dec 27 04:59:48.407361226 (517h30m48.278444261s) [noise=true , address=145.79.227.23:4546 , piece_id=SDALLDC7VDZIBLXEVZYEAFMSNR3BHN6U3M3B6XHWYJMAE2GODXEA, seg_pos={0 0}, upload=454047]: Done uploading piece. err: <nil> Dec 27 04:59:48.408696699 (517h30m48.279779734s) [noise=true , address=145.79.227.48:4302 , piece_id=T2QFAB5QKNYUCPQ6N5BY7E2CL3TNYE7VZUSIOLAPAKRQCW6WA2GA, seg_pos={0 0}, upload=454047]: Done uploading piece. err: <nil> Dec 27 04:59:48.417150977 (517h30m48.288234012s) [noise=true , address=145.79.227.47:4266 , piece_id=A3AO6DOOC7RO4CZWA3OCXWHJZEWYBOSWZN42AHTYLUBW26HRGPOA, seg_pos={0 0}, upload=454047]: Done uploading piece. err: <nil> Dec 27 04:59:48.424662841 (517h30m48.295745886s) [noise=true , address=145.79.227.28:4725 , piece_id=U3WTJJMPYMYJGOAUOOTVFCSSQHSPQWZ7CLCF4O5I5VKJI2MWSNMQ, seg_pos={0 0}, upload=454047]: Done uploading piece. err: <nil> Dec 27 04:59:48.426624607 (517h30m48.297707642s) [noise=true , address=145.79.227.6:4259 , piece_id=3MDQTKVA7EQGRRKQBXNTSW4ESHWM6343HDGYE2MOIK2Q3A3DJZUQ, seg_pos={0 0}, upload=454047]: Done uploading piece. err: <nil> Dec 27 04:59:48.428890933 (517h30m48.299973968s) [noise=true , address=145.79.227.42:4121 , piece_id=JYA7LUS7PDUML2XQ3TKFG7LVBSQS6U2TOUZQZDJHBONCQK2OQ7JA, seg_pos={0 0}, upload=454047]: Done uploading piece. err: <nil> Dec 27 04:59:48.432133296 (517h30m48.303216331s) [noise=true , address=145.79.227.37:4565 , piece_id=NQRYLDU72RS7LPYKSL34EDJJBQENXU72QCOBU3TZZCRW22KCXDCQ, seg_pos={0 0}, upload=454047]: Done uploading piece. err: <nil> Dec 27 04:59:48.435291590 (517h30m48.306374635s) [noise=true , address=145.79.227.3:4151 , piece_id=T5BZO7VAI5IQQA3W4PIEM6VCYS52KI44WB7ZE7KRTJXORZ5V5ACA, seg_pos={0 0}, upload=454047]: Done uploading piece. err: <nil> Dec 27 04:59:48.608504507 (517h30m48.479587542s) [noise=true , address=77.90.31.15:4700 , piece_id=PNSLBIFTV7OENCOY2VRBFK3XPV2SWITSRUIPFBZ7LLO5KZQVLJYQ, seg_pos={0 0}, upload=454047]: Done uploading piece. err: <nil> Dec 27 04:59:48.624416867 (517h30m48.495499902s) [noise=true , address=45.91.116.5:4299 , piece_id=FWUY6XRZ34UZY44Q4Q3TEWRH5PS5WATSSLMYC2VI6LPEFV6XJZCA, seg_pos={0 0}, upload=454047]: Done uploading piece. err: <nil> Dec 27 04:59:48.659913960 (517h30m48.530996995s) [noise=true , address=45.91.116.12:4227 , piece_id=2RAUNHDOPUIWT57I3VFZTFLTDFQG2QL6WX5YKFQKFGENSBGKJJBA, seg_pos={0 0}, upload=454047]: Done uploading piece. err: <nil> Dec 27 04:59:48.659933306 (517h30m48.531016341s) [seg_pos={0 0}, upload=454047]: Segment reached optimal threshold of 13 pieces. Dec 27 04:59:48.660109081 (517h30m48.531192116s) [noise=true , address=77.90.31.11:4556 , piece_id=6AGQQJEOZD5CYRDHSCG6IGIZBDKFEGEHTFJ2TGJFQ3ODECT367EQ, seg_pos={0 0}, upload=454047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 27 04:59:48.660115173 (517h30m48.531198218s) [noise=true , address=77.90.31.11:4556 , piece_id=6AGQQJEOZD5CYRDHSCG6IGIZBDKFEGEHTFJ2TGJFQ3ODECT367EQ, seg_pos={0 0}, upload=454047]: Long tail cancellation due to optimal threshold Dec 27 04:59:48.660318710 (517h30m48.531401755s) [noise=true , address=77.90.31.16:4735 , piece_id=EMVLUIFA4LAIC74HUQLQELPIYJVJ7NALJPO7TP7GODGQ3LNFTFIQ, seg_pos={0 0}, upload=454047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 27 04:59:48.660336524 (517h30m48.531419559s) [noise=true , address=77.90.31.16:4735 , piece_id=EMVLUIFA4LAIC74HUQLQELPIYJVJ7NALJPO7TP7GODGQ3LNFTFIQ, seg_pos={0 0}, upload=454047]: Long tail cancellation due to optimal threshold Dec 27 04:59:48.660426535 (517h30m48.53150957s) [noise=true , address=45.91.116.13:4153 , piece_id=HJ5XOXJFEI5JNLYK4TSZFZBZZAWR7NBVTA2EVMKSECEOCQA377FQ, seg_pos={0 0}, upload=454047]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 27 04:59:48.660432257 (517h30m48.531515302s) [noise=true , address=45.91.116.13:4153 , piece_id=HJ5XOXJFEI5JNLYK4TSZFZBZZAWR7NBVTA2EVMKSECEOCQA377FQ, seg_pos={0 0}, upload=454047]: Long tail cancellation due to optimal threshold Dec 27 04:59:48.660448948 (517h30m48.531531983s) [seg_pos={0 0}, upload=454047]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:59:48.660536856 (517h30m48.531619891s) [upload=454047]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:59:48.660543709 (517h30m48.531626744s) [upload=454047]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:59:48.950791860 (517h30m48.821874905s) [upload=454047]: Done uploading segments... Dec 27 04:59:48.950810295 (517h30m48.82189333s) [upload=454047]: Upload finished. err: <nil> Duration: 30.003243615s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 3.492728684s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 04:59:56 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.492754122s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X......................X............] duration: 47.190113166s trace: https://jaeger.dev.storj.io/trace/60d70ba2fe18218 FAIL (2025-12-27 04:59:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 4 (517h19m18.685148379s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814531152 (517h19m18.685614187s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814852964 (517h19m18.685936009s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815170900 (517h19m18.686253945s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815486661 (517h19m18.686569706s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815835786 (517h19m18.686918831s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.816207564 (517h19m18.687290609s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818225116 (517h19m18.689308151s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818325547 (517h19m18.689408592s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.821132431 (517h19m18.692215466s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.822845684 (517h19m18.693928719s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.824299251 (517h19m18.695382296s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.825754102 (517h19m18.696837147s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.827122548 (517h19m18.698205593s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.828453653 (517h19m18.699536688s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:20.114191076 (517h19m19.985274121s) [upload=453889]: Next returned nil segment Dec 27 04:48:20.114206816 (517h19m19.985289851s) [upload=453889]: Waiting for error group managing segments... Dec 27 04:48:20.444209056 (517h19m20.315292102s) [noise=true , address=64.49.13.154:4929 , piece_id=2EXG7BEQ6NKAG3TNAH3TZ3K5ZI4KNOQYF3OEZMFWVX6BBMWNVXUQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.474597952 (517h19m20.345680987s) [noise=true , address=64.49.13.27:4964 , piece_id=XAROGTY2JLVFUVDYT7N7ZCABVYDMXA45NZF36NBAV4H72M3PJSCA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.478601025 (517h19m20.34968406s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.888413067 (517h19m20.759496102s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.930510167 (517h19m20.801593212s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.007760305 (517h19m20.87884334s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.041631362 (517h19m20.912714407s) [noise=true , address=77.90.31.14:4691 , piece_id=HIFLW7CWEYK6A6BZAYPDZLIUNA26IFXSFNVYV2UK4VPKEWYUMYJA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.052961556 (517h19m20.924044601s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:23.606703638 (517h19m23.477786673s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.046520345 (517h19m25.91760338s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.518771011 (517h19m26.389854046s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.589256778 (517h19m26.460339823s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:27.044280620 (517h19m26.915363655s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789125901 (517h19m36.660208936s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789145728 (517h19m36.660228764s) [seg_pos={0 0}, upload=453889]: Segment reached optimal threshold of 14 pieces. Dec 27 04:48:36.789615964 (517h19m36.660699009s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: read tcp 188.166.225.170:50246->145.79.227.40:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789624180 (517h19m36.660707215s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789761231 (517h19m36.660844276s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: read tcp 188.166.225.170:45976->145.79.227.44:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789766831 (517h19m36.660849866s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789924742 (517h19m36.661007788s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51): protocol: read tcp 188.166.225.170:43134->145.79.227.43:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789933699 (517h19m36.661016734s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790128380 (517h19m36.661211425s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1): protocol: read tcp 188.166.225.170:46462->145.79.227.25:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.790137838 (517h19m36.661220873s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790156633 (517h19m36.661239679s) [seg_pos={0 0}, upload=453889]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:48:36.790241626 (517h19m36.661324661s) [upload=453889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:48:36.790246826 (517h19m36.661329861s) [upload=453889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:48:37.084198870 (517h19m36.955281915s) [upload=453889]: Done uploading segments... Dec 27 04:48:37.084219929 (517h19m36.955302974s) [upload=453889]: Upload finished. err: <nil> Duration: 47.190138724s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 3.453951253s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 04:59:08 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.453979486s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X......................X...........] duration: 50.783721333s trace: https://jaeger.dev.storj.io/trace/60d70ba2fe18218 FAIL (2025-12-27 04:59:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 4 (517h19m18.685148379s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814531152 (517h19m18.685614187s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814852964 (517h19m18.685936009s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815170900 (517h19m18.686253945s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815486661 (517h19m18.686569706s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815835786 (517h19m18.686918831s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.816207564 (517h19m18.687290609s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818225116 (517h19m18.689308151s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818325547 (517h19m18.689408592s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.821132431 (517h19m18.692215466s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.822845684 (517h19m18.693928719s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.824299251 (517h19m18.695382296s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.825754102 (517h19m18.696837147s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.827122548 (517h19m18.698205593s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.828453653 (517h19m18.699536688s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:20.114191076 (517h19m19.985274121s) [upload=453889]: Next returned nil segment Dec 27 04:48:20.114206816 (517h19m19.985289851s) [upload=453889]: Waiting for error group managing segments... Dec 27 04:48:20.444209056 (517h19m20.315292102s) [noise=true , address=64.49.13.154:4929 , piece_id=2EXG7BEQ6NKAG3TNAH3TZ3K5ZI4KNOQYF3OEZMFWVX6BBMWNVXUQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.474597952 (517h19m20.345680987s) [noise=true , address=64.49.13.27:4964 , piece_id=XAROGTY2JLVFUVDYT7N7ZCABVYDMXA45NZF36NBAV4H72M3PJSCA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.478601025 (517h19m20.34968406s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.888413067 (517h19m20.759496102s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.930510167 (517h19m20.801593212s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.007760305 (517h19m20.87884334s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.041631362 (517h19m20.912714407s) [noise=true , address=77.90.31.14:4691 , piece_id=HIFLW7CWEYK6A6BZAYPDZLIUNA26IFXSFNVYV2UK4VPKEWYUMYJA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.052961556 (517h19m20.924044601s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:23.606703638 (517h19m23.477786673s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.046520345 (517h19m25.91760338s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.518771011 (517h19m26.389854046s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.589256778 (517h19m26.460339823s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:27.044280620 (517h19m26.915363655s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789125901 (517h19m36.660208936s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789145728 (517h19m36.660228764s) [seg_pos={0 0}, upload=453889]: Segment reached optimal threshold of 14 pieces. Dec 27 04:48:36.789615964 (517h19m36.660699009s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: read tcp 188.166.225.170:50246->145.79.227.40:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789624180 (517h19m36.660707215s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789761231 (517h19m36.660844276s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: read tcp 188.166.225.170:45976->145.79.227.44:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789766831 (517h19m36.660849866s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789924742 (517h19m36.661007788s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51): protocol: read tcp 188.166.225.170:43134->145.79.227.43:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789933699 (517h19m36.661016734s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790128380 (517h19m36.661211425s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1): protocol: read tcp 188.166.225.170:46462->145.79.227.25:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.790137838 (517h19m36.661220873s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790156633 (517h19m36.661239679s) [seg_pos={0 0}, upload=453889]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:48:36.790241626 (517h19m36.661324661s) [upload=453889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:48:36.790246826 (517h19m36.661329861s) [upload=453889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:48:37.084198870 (517h19m36.955281915s) [upload=453889]: Done uploading segments... Dec 27 04:48:37.084219929 (517h19m36.955302974s) [upload=453889]: Upload finished. err: <nil> Duration: 50.783749226s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [...XXXX.X.X....X...XX.XXX..XXX..XXXX...XX...X.XX....XXXX..X.] duration: 13.818986331s trace: https://jaeger.dev.storj.io/trace/6390b17b5edf4ccc FAIL (2025-12-27 04:58:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 04:57:28.726777563 (517h28m28.597860598s) [upload=454018]: Starting upload Dec 27 04:57:28.726874217 (517h28m28.597957262s) [upload=454018]: Done starting upload Dec 27 04:57:28.727000007 (517h28m28.598083052s) [upload=454018]: Uploading segments... Dec 27 04:57:28.727237108 (517h28m28.598320153s) [upload=454018]: Got next segment. Inline: false Dec 27 04:57:28.727279439 (517h28m28.598362474s) [upload=454018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:57:28.727285130 (517h28m28.598368165s) [upload=454018]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:57:28.983209413 (517h28m28.854292468s) [seg_pos={0 0}, upload=454018]: Begin upload segment... Dec 27 04:57:28.983262725 (517h28m28.85434577s) [seg_pos={0 0}, upload=454018]: Done begin upload segment. Dec 27 04:57:28.983296289 (517h28m28.854379334s) [noise=true , address=45.91.116.14:4192 , piece_id=25VV4LWEAHW6IV7DZYKMD7QSIJHCNTT6NAHC64JG3FS64LK754PA, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.983334321 (517h28m28.854417366s) [noise=true , address=145.79.227.42:4121 , piece_id=NITUZXHONLJZV7ABSUXOBOFJP4FWIZBPIOTMXKR7XAOY7CJYUC6Q, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.984019315 (517h28m28.85510235s) [noise=true , address=145.79.227.15:4874 , piece_id=XUTKF5G5DMNIRKEVQODAVRWLBW26NRVE5ZSVBFP6RZDI63POZC2A, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.984079700 (517h28m28.855162745s) [noise=true , address=145.79.227.11:4729 , piece_id=OUJQBQVK62YAW4Y57B3VUVLY3WX6HO4PAGV56SMZRNTCXYFEX37Q, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.984630639 (517h28m28.855713674s) [noise=true , address=145.79.227.36:4519 , piece_id=SYCUWEDK3NWWBY6AWJCRUUF3W6BT7YUTFRJK52PUTLABNPE22LGQ, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.985234529 (517h28m28.856317574s) [noise=true , address=45.91.116.7:4009 , piece_id=AVHNEBFAHCDL5G5KWOS7KAPIYXAMHR2CMOKIGOOZAGBZQS2AFTSA, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.985608100 (517h28m28.856691136s) [noise=true , address=145.79.227.10:4403 , piece_id=TKP2W5Q35SVHFSMFNL6ZLF2SLWIOUD6G2C6C4IYOT6CJZHMIMKZQ, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.985825514 (517h28m28.856908559s) [noise=true , address=145.79.227.38:4979 , piece_id=FIVEH7ZHEBUH22P2K4UJB7E4TNJVKDYDLPO2WKFXXF35W2QXR2HA, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.986017941 (517h28m28.857100976s) [noise=true , address=77.90.31.18:4808 , piece_id=O5MCFVIUSYB725F4QH6JARKMXOSQAXSMEXW36SJBGWFBLP4WX3IQ, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.986361275 (517h28m28.85744432s) [noise=true , address=145.79.227.45:4230 , piece_id=5FQGPG6WJTEXZ5TFWJSDEK7CGXH27YBXW44XRNJJWDOL3O72OJ4A, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.986373278 (517h28m28.857456313s) [noise=true , address=45.91.116.11:4120 , piece_id=CNCVXNGV32IN3C5FLBNNQSDYWHNIJQBBYJU4PMXPLXSNZ76GG7PQ, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.986824908 (517h28m28.857907943s) [noise=true , address=45.91.116.5:4298 , piece_id=FNTZRWTBJNRNRXV3TWBHKC3FHDUYVALVIUOUBBY3WEGBNSLZDK3A, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.987025450 (517h28m28.858108495s) [noise=true , address=77.90.31.14:4664 , piece_id=HMZNH5YGI77YMITYH7PI5NRWXBK5MYKF4IPN7FWKXA2XJYG647KQ, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.987163753 (517h28m28.858246798s) [noise=true , address=145.79.227.13:4802 , piece_id=SBGS2QNO54CJSIZKT6RRVKXYOL32J2CPIATHKABOJST5YCOCEQRA, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.987737024 (517h28m28.858820059s) [noise=true , address=145.79.227.46:4601 , piece_id=67MAU6L6IF2DULXQTSWXGCDI5SJWXM4A75CLNWRLOPWBI62R62IA, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.987800986 (517h28m28.858884031s) [noise=true , address=77.90.31.17:4770 , piece_id=TSZKXNQZ2MXTKCPR6VLNMFVMRLETL2Y5EWZ57MCTK4RBQTZU4D7A, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:30.599796542 (517h28m30.470879577s) [upload=454018]: Next returned nil segment Dec 27 04:57:30.599812842 (517h28m30.470895877s) [upload=454018]: Waiting for error group managing segments... Dec 27 04:57:30.985521317 (517h28m30.856604353s) [noise=true , address=145.79.227.42:4121 , piece_id=NITUZXHONLJZV7ABSUXOBOFJP4FWIZBPIOTMXKR7XAOY7CJYUC6Q, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:30.997362936 (517h28m30.868445971s) [noise=true , address=145.79.227.36:4519 , piece_id=SYCUWEDK3NWWBY6AWJCRUUF3W6BT7YUTFRJK52PUTLABNPE22LGQ, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.002520557 (517h28m30.873603592s) [noise=true , address=145.79.227.38:4979 , piece_id=FIVEH7ZHEBUH22P2K4UJB7E4TNJVKDYDLPO2WKFXXF35W2QXR2HA, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.005987178 (517h28m30.877070223s) [noise=true , address=145.79.227.46:4601 , piece_id=67MAU6L6IF2DULXQTSWXGCDI5SJWXM4A75CLNWRLOPWBI62R62IA, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.009730596 (517h28m30.880813641s) [noise=true , address=145.79.227.11:4729 , piece_id=OUJQBQVK62YAW4Y57B3VUVLY3WX6HO4PAGV56SMZRNTCXYFEX37Q, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.009782414 (517h28m30.880865459s) [noise=true , address=145.79.227.15:4874 , piece_id=XUTKF5G5DMNIRKEVQODAVRWLBW26NRVE5ZSVBFP6RZDI63POZC2A, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.015590273 (517h28m30.886673308s) [noise=true , address=145.79.227.10:4403 , piece_id=TKP2W5Q35SVHFSMFNL6ZLF2SLWIOUD6G2C6C4IYOT6CJZHMIMKZQ, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.020506705 (517h28m30.89158974s) [noise=true , address=145.79.227.45:4230 , piece_id=5FQGPG6WJTEXZ5TFWJSDEK7CGXH27YBXW44XRNJJWDOL3O72OJ4A, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.020525381 (517h28m30.891608516s) [noise=true , address=145.79.227.13:4802 , piece_id=SBGS2QNO54CJSIZKT6RRVKXYOL32J2CPIATHKABOJST5YCOCEQRA, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.239196067 (517h28m31.110279112s) [noise=true , address=77.90.31.18:4808 , piece_id=O5MCFVIUSYB725F4QH6JARKMXOSQAXSMEXW36SJBGWFBLP4WX3IQ, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.265170878 (517h28m31.136253923s) [noise=true , address=77.90.31.17:4770 , piece_id=TSZKXNQZ2MXTKCPR6VLNMFVMRLETL2Y5EWZ57MCTK4RBQTZU4D7A, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.269304979 (517h28m31.140388024s) [noise=true , address=77.90.31.14:4664 , piece_id=HMZNH5YGI77YMITYH7PI5NRWXBK5MYKF4IPN7FWKXA2XJYG647KQ, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.312355986 (517h28m31.183439031s) [noise=true , address=45.91.116.7:4009 , piece_id=AVHNEBFAHCDL5G5KWOS7KAPIYXAMHR2CMOKIGOOZAGBZQS2AFTSA, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.312374862 (517h28m31.183457897s) [seg_pos={0 0}, upload=454018]: Segment reached optimal threshold of 13 pieces. Dec 27 04:57:31.312611743 (517h28m31.183694778s) [noise=true , address=45.91.116.11:4120 , piece_id=CNCVXNGV32IN3C5FLBNNQSDYWHNIJQBBYJU4PMXPLXSNZ76GG7PQ, seg_pos={0 0}, upload=454018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 04:57:31.312617914 (517h28m31.18370095s) [noise=true , address=45.91.116.11:4120 , piece_id=CNCVXNGV32IN3C5FLBNNQSDYWHNIJQBBYJU4PMXPLXSNZ76GG7PQ, seg_pos={0 0}, upload=454018]: Long tail cancellation due to optimal threshold Dec 27 04:57:31.312669383 (517h28m31.183752418s) [noise=true , address=45.91.116.5:4298 , piece_id=FNTZRWTBJNRNRXV3TWBHKC3FHDUYVALVIUOUBBY3WEGBNSLZDK3A, seg_pos={0 0}, upload=454018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:52858->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:57:31.312687627 (517h28m31.183770662s) [noise=true , address=45.91.116.5:4298 , piece_id=FNTZRWTBJNRNRXV3TWBHKC3FHDUYVALVIUOUBBY3WEGBNSLZDK3A, seg_pos={0 0}, upload=454018]: Long tail cancellation due to optimal threshold Dec 27 04:57:31.312744526 (517h28m31.183827571s) [noise=true , address=45.91.116.14:4192 , piece_id=25VV4LWEAHW6IV7DZYKMD7QSIJHCNTT6NAHC64JG3FS64LK754PA, seg_pos={0 0}, upload=454018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 27 04:57:31.312750507 (517h28m31.183833542s) [noise=true , address=45.91.116.14:4192 , piece_id=25VV4LWEAHW6IV7DZYKMD7QSIJHCNTT6NAHC64JG3FS64LK754PA, seg_pos={0 0}, upload=454018]: Long tail cancellation due to optimal threshold Dec 27 04:57:31.312766729 (517h28m31.183849764s) [seg_pos={0 0}, upload=454018]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:57:31.312830609 (517h28m31.183913654s) [upload=454018]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:57:31.312834989 (517h28m31.183918024s) [upload=454018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:57:31.612010370 (517h28m31.483093425s) [upload=454018]: Done uploading segments... Dec 27 04:57:31.612035938 (517h28m31.483118983s) [upload=454018]: Upload finished. err: <nil> Duration: 13.819025397s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 3.476824307s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 04:58:30 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.476854154s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 3.697907879s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 04:58:08 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.698042585s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [....XXXX.X.X....X...XX.XXX..XXX..XXXX...XX...X.XX....XXXX..X] duration: 30.003871602s trace: https://jaeger.dev.storj.io/trace/6390b17b5edf4ccc FAIL (2025-12-27 04:57:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 04:57:28.726777563 (517h28m28.597860598s) [upload=454018]: Starting upload Dec 27 04:57:28.726874217 (517h28m28.597957262s) [upload=454018]: Done starting upload Dec 27 04:57:28.727000007 (517h28m28.598083052s) [upload=454018]: Uploading segments... Dec 27 04:57:28.727237108 (517h28m28.598320153s) [upload=454018]: Got next segment. Inline: false Dec 27 04:57:28.727279439 (517h28m28.598362474s) [upload=454018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:57:28.727285130 (517h28m28.598368165s) [upload=454018]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:57:28.983209413 (517h28m28.854292468s) [seg_pos={0 0}, upload=454018]: Begin upload segment... Dec 27 04:57:28.983262725 (517h28m28.85434577s) [seg_pos={0 0}, upload=454018]: Done begin upload segment. Dec 27 04:57:28.983296289 (517h28m28.854379334s) [noise=true , address=45.91.116.14:4192 , piece_id=25VV4LWEAHW6IV7DZYKMD7QSIJHCNTT6NAHC64JG3FS64LK754PA, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.983334321 (517h28m28.854417366s) [noise=true , address=145.79.227.42:4121 , piece_id=NITUZXHONLJZV7ABSUXOBOFJP4FWIZBPIOTMXKR7XAOY7CJYUC6Q, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.984019315 (517h28m28.85510235s) [noise=true , address=145.79.227.15:4874 , piece_id=XUTKF5G5DMNIRKEVQODAVRWLBW26NRVE5ZSVBFP6RZDI63POZC2A, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.984079700 (517h28m28.855162745s) [noise=true , address=145.79.227.11:4729 , piece_id=OUJQBQVK62YAW4Y57B3VUVLY3WX6HO4PAGV56SMZRNTCXYFEX37Q, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.984630639 (517h28m28.855713674s) [noise=true , address=145.79.227.36:4519 , piece_id=SYCUWEDK3NWWBY6AWJCRUUF3W6BT7YUTFRJK52PUTLABNPE22LGQ, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.985234529 (517h28m28.856317574s) [noise=true , address=45.91.116.7:4009 , piece_id=AVHNEBFAHCDL5G5KWOS7KAPIYXAMHR2CMOKIGOOZAGBZQS2AFTSA, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.985608100 (517h28m28.856691136s) [noise=true , address=145.79.227.10:4403 , piece_id=TKP2W5Q35SVHFSMFNL6ZLF2SLWIOUD6G2C6C4IYOT6CJZHMIMKZQ, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.985825514 (517h28m28.856908559s) [noise=true , address=145.79.227.38:4979 , piece_id=FIVEH7ZHEBUH22P2K4UJB7E4TNJVKDYDLPO2WKFXXF35W2QXR2HA, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.986017941 (517h28m28.857100976s) [noise=true , address=77.90.31.18:4808 , piece_id=O5MCFVIUSYB725F4QH6JARKMXOSQAXSMEXW36SJBGWFBLP4WX3IQ, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.986361275 (517h28m28.85744432s) [noise=true , address=145.79.227.45:4230 , piece_id=5FQGPG6WJTEXZ5TFWJSDEK7CGXH27YBXW44XRNJJWDOL3O72OJ4A, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.986373278 (517h28m28.857456313s) [noise=true , address=45.91.116.11:4120 , piece_id=CNCVXNGV32IN3C5FLBNNQSDYWHNIJQBBYJU4PMXPLXSNZ76GG7PQ, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.986824908 (517h28m28.857907943s) [noise=true , address=45.91.116.5:4298 , piece_id=FNTZRWTBJNRNRXV3TWBHKC3FHDUYVALVIUOUBBY3WEGBNSLZDK3A, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.987025450 (517h28m28.858108495s) [noise=true , address=77.90.31.14:4664 , piece_id=HMZNH5YGI77YMITYH7PI5NRWXBK5MYKF4IPN7FWKXA2XJYG647KQ, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.987163753 (517h28m28.858246798s) [noise=true , address=145.79.227.13:4802 , piece_id=SBGS2QNO54CJSIZKT6RRVKXYOL32J2CPIATHKABOJST5YCOCEQRA, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.987737024 (517h28m28.858820059s) [noise=true , address=145.79.227.46:4601 , piece_id=67MAU6L6IF2DULXQTSWXGCDI5SJWXM4A75CLNWRLOPWBI62R62IA, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:28.987800986 (517h28m28.858884031s) [noise=true , address=77.90.31.17:4770 , piece_id=TSZKXNQZ2MXTKCPR6VLNMFVMRLETL2Y5EWZ57MCTK4RBQTZU4D7A, seg_pos={0 0}, upload=454018]: Uploading piece... Dec 27 04:57:30.599796542 (517h28m30.470879577s) [upload=454018]: Next returned nil segment Dec 27 04:57:30.599812842 (517h28m30.470895877s) [upload=454018]: Waiting for error group managing segments... Dec 27 04:57:30.985521317 (517h28m30.856604353s) [noise=true , address=145.79.227.42:4121 , piece_id=NITUZXHONLJZV7ABSUXOBOFJP4FWIZBPIOTMXKR7XAOY7CJYUC6Q, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:30.997362936 (517h28m30.868445971s) [noise=true , address=145.79.227.36:4519 , piece_id=SYCUWEDK3NWWBY6AWJCRUUF3W6BT7YUTFRJK52PUTLABNPE22LGQ, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.002520557 (517h28m30.873603592s) [noise=true , address=145.79.227.38:4979 , piece_id=FIVEH7ZHEBUH22P2K4UJB7E4TNJVKDYDLPO2WKFXXF35W2QXR2HA, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.005987178 (517h28m30.877070223s) [noise=true , address=145.79.227.46:4601 , piece_id=67MAU6L6IF2DULXQTSWXGCDI5SJWXM4A75CLNWRLOPWBI62R62IA, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.009730596 (517h28m30.880813641s) [noise=true , address=145.79.227.11:4729 , piece_id=OUJQBQVK62YAW4Y57B3VUVLY3WX6HO4PAGV56SMZRNTCXYFEX37Q, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.009782414 (517h28m30.880865459s) [noise=true , address=145.79.227.15:4874 , piece_id=XUTKF5G5DMNIRKEVQODAVRWLBW26NRVE5ZSVBFP6RZDI63POZC2A, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.015590273 (517h28m30.886673308s) [noise=true , address=145.79.227.10:4403 , piece_id=TKP2W5Q35SVHFSMFNL6ZLF2SLWIOUD6G2C6C4IYOT6CJZHMIMKZQ, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.020506705 (517h28m30.89158974s) [noise=true , address=145.79.227.45:4230 , piece_id=5FQGPG6WJTEXZ5TFWJSDEK7CGXH27YBXW44XRNJJWDOL3O72OJ4A, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.020525381 (517h28m30.891608516s) [noise=true , address=145.79.227.13:4802 , piece_id=SBGS2QNO54CJSIZKT6RRVKXYOL32J2CPIATHKABOJST5YCOCEQRA, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.239196067 (517h28m31.110279112s) [noise=true , address=77.90.31.18:4808 , piece_id=O5MCFVIUSYB725F4QH6JARKMXOSQAXSMEXW36SJBGWFBLP4WX3IQ, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.265170878 (517h28m31.136253923s) [noise=true , address=77.90.31.17:4770 , piece_id=TSZKXNQZ2MXTKCPR6VLNMFVMRLETL2Y5EWZ57MCTK4RBQTZU4D7A, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.269304979 (517h28m31.140388024s) [noise=true , address=77.90.31.14:4664 , piece_id=HMZNH5YGI77YMITYH7PI5NRWXBK5MYKF4IPN7FWKXA2XJYG647KQ, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.312355986 (517h28m31.183439031s) [noise=true , address=45.91.116.7:4009 , piece_id=AVHNEBFAHCDL5G5KWOS7KAPIYXAMHR2CMOKIGOOZAGBZQS2AFTSA, seg_pos={0 0}, upload=454018]: Done uploading piece. err: <nil> Dec 27 04:57:31.312374862 (517h28m31.183457897s) [seg_pos={0 0}, upload=454018]: Segment reached optimal threshold of 13 pieces. Dec 27 04:57:31.312611743 (517h28m31.183694778s) [noise=true , address=45.91.116.11:4120 , piece_id=CNCVXNGV32IN3C5FLBNNQSDYWHNIJQBBYJU4PMXPLXSNZ76GG7PQ, seg_pos={0 0}, upload=454018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 04:57:31.312617914 (517h28m31.18370095s) [noise=true , address=45.91.116.11:4120 , piece_id=CNCVXNGV32IN3C5FLBNNQSDYWHNIJQBBYJU4PMXPLXSNZ76GG7PQ, seg_pos={0 0}, upload=454018]: Long tail cancellation due to optimal threshold Dec 27 04:57:31.312669383 (517h28m31.183752418s) [noise=true , address=45.91.116.5:4298 , piece_id=FNTZRWTBJNRNRXV3TWBHKC3FHDUYVALVIUOUBBY3WEGBNSLZDK3A, seg_pos={0 0}, upload=454018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:52858->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:57:31.312687627 (517h28m31.183770662s) [noise=true , address=45.91.116.5:4298 , piece_id=FNTZRWTBJNRNRXV3TWBHKC3FHDUYVALVIUOUBBY3WEGBNSLZDK3A, seg_pos={0 0}, upload=454018]: Long tail cancellation due to optimal threshold Dec 27 04:57:31.312744526 (517h28m31.183827571s) [noise=true , address=45.91.116.14:4192 , piece_id=25VV4LWEAHW6IV7DZYKMD7QSIJHCNTT6NAHC64JG3FS64LK754PA, seg_pos={0 0}, upload=454018]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 27 04:57:31.312750507 (517h28m31.183833542s) [noise=true , address=45.91.116.14:4192 , piece_id=25VV4LWEAHW6IV7DZYKMD7QSIJHCNTT6NAHC64JG3FS64LK754PA, seg_pos={0 0}, upload=454018]: Long tail cancellation due to optimal threshold Dec 27 04:57:31.312766729 (517h28m31.183849764s) [seg_pos={0 0}, upload=454018]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:57:31.312830609 (517h28m31.183913654s) [upload=454018]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:57:31.312834989 (517h28m31.183918024s) [upload=454018]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:57:31.612010370 (517h28m31.483093425s) [upload=454018]: Done uploading segments... Dec 27 04:57:31.612035938 (517h28m31.483118983s) [upload=454018]: Upload finished. err: <nil> Duration: 30.003893703s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X......................X..........] duration: 37.788932991s trace: https://jaeger.dev.storj.io/trace/60d70ba2fe18218 FAIL (2025-12-27 04:57:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 4 (517h19m18.685148379s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814531152 (517h19m18.685614187s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814852964 (517h19m18.685936009s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815170900 (517h19m18.686253945s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815486661 (517h19m18.686569706s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815835786 (517h19m18.686918831s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.816207564 (517h19m18.687290609s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818225116 (517h19m18.689308151s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818325547 (517h19m18.689408592s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.821132431 (517h19m18.692215466s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.822845684 (517h19m18.693928719s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.824299251 (517h19m18.695382296s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.825754102 (517h19m18.696837147s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.827122548 (517h19m18.698205593s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.828453653 (517h19m18.699536688s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:20.114191076 (517h19m19.985274121s) [upload=453889]: Next returned nil segment Dec 27 04:48:20.114206816 (517h19m19.985289851s) [upload=453889]: Waiting for error group managing segments... Dec 27 04:48:20.444209056 (517h19m20.315292102s) [noise=true , address=64.49.13.154:4929 , piece_id=2EXG7BEQ6NKAG3TNAH3TZ3K5ZI4KNOQYF3OEZMFWVX6BBMWNVXUQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.474597952 (517h19m20.345680987s) [noise=true , address=64.49.13.27:4964 , piece_id=XAROGTY2JLVFUVDYT7N7ZCABVYDMXA45NZF36NBAV4H72M3PJSCA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.478601025 (517h19m20.34968406s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.888413067 (517h19m20.759496102s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.930510167 (517h19m20.801593212s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.007760305 (517h19m20.87884334s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.041631362 (517h19m20.912714407s) [noise=true , address=77.90.31.14:4691 , piece_id=HIFLW7CWEYK6A6BZAYPDZLIUNA26IFXSFNVYV2UK4VPKEWYUMYJA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.052961556 (517h19m20.924044601s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:23.606703638 (517h19m23.477786673s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.046520345 (517h19m25.91760338s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.518771011 (517h19m26.389854046s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.589256778 (517h19m26.460339823s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:27.044280620 (517h19m26.915363655s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789125901 (517h19m36.660208936s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789145728 (517h19m36.660228764s) [seg_pos={0 0}, upload=453889]: Segment reached optimal threshold of 14 pieces. Dec 27 04:48:36.789615964 (517h19m36.660699009s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: read tcp 188.166.225.170:50246->145.79.227.40:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789624180 (517h19m36.660707215s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789761231 (517h19m36.660844276s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: read tcp 188.166.225.170:45976->145.79.227.44:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789766831 (517h19m36.660849866s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789924742 (517h19m36.661007788s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51): protocol: read tcp 188.166.225.170:43134->145.79.227.43:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789933699 (517h19m36.661016734s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790128380 (517h19m36.661211425s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1): protocol: read tcp 188.166.225.170:46462->145.79.227.25:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.790137838 (517h19m36.661220873s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790156633 (517h19m36.661239679s) [seg_pos={0 0}, upload=453889]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:48:36.790241626 (517h19m36.661324661s) [upload=453889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:48:36.790246826 (517h19m36.661329861s) [upload=453889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:48:37.084198870 (517h19m36.955281915s) [upload=453889]: Done uploading segments... Dec 27 04:48:37.084219929 (517h19m36.955302974s) [upload=453889]: Upload finished. err: <nil> Duration: 37.788947779s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 3.507762088s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 04:57:48 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.507784852s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X......................X.........] duration: 42.028824034s trace: https://jaeger.dev.storj.io/trace/60d70ba2fe18218 FAIL (2025-12-27 04:57:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 4 (517h19m18.685148379s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814531152 (517h19m18.685614187s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814852964 (517h19m18.685936009s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815170900 (517h19m18.686253945s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815486661 (517h19m18.686569706s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815835786 (517h19m18.686918831s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.816207564 (517h19m18.687290609s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818225116 (517h19m18.689308151s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818325547 (517h19m18.689408592s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.821132431 (517h19m18.692215466s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.822845684 (517h19m18.693928719s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.824299251 (517h19m18.695382296s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.825754102 (517h19m18.696837147s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.827122548 (517h19m18.698205593s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.828453653 (517h19m18.699536688s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:20.114191076 (517h19m19.985274121s) [upload=453889]: Next returned nil segment Dec 27 04:48:20.114206816 (517h19m19.985289851s) [upload=453889]: Waiting for error group managing segments... Dec 27 04:48:20.444209056 (517h19m20.315292102s) [noise=true , address=64.49.13.154:4929 , piece_id=2EXG7BEQ6NKAG3TNAH3TZ3K5ZI4KNOQYF3OEZMFWVX6BBMWNVXUQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.474597952 (517h19m20.345680987s) [noise=true , address=64.49.13.27:4964 , piece_id=XAROGTY2JLVFUVDYT7N7ZCABVYDMXA45NZF36NBAV4H72M3PJSCA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.478601025 (517h19m20.34968406s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.888413067 (517h19m20.759496102s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.930510167 (517h19m20.801593212s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.007760305 (517h19m20.87884334s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.041631362 (517h19m20.912714407s) [noise=true , address=77.90.31.14:4691 , piece_id=HIFLW7CWEYK6A6BZAYPDZLIUNA26IFXSFNVYV2UK4VPKEWYUMYJA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.052961556 (517h19m20.924044601s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:23.606703638 (517h19m23.477786673s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.046520345 (517h19m25.91760338s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.518771011 (517h19m26.389854046s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.589256778 (517h19m26.460339823s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:27.044280620 (517h19m26.915363655s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789125901 (517h19m36.660208936s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789145728 (517h19m36.660228764s) [seg_pos={0 0}, upload=453889]: Segment reached optimal threshold of 14 pieces. Dec 27 04:48:36.789615964 (517h19m36.660699009s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: read tcp 188.166.225.170:50246->145.79.227.40:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789624180 (517h19m36.660707215s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789761231 (517h19m36.660844276s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: read tcp 188.166.225.170:45976->145.79.227.44:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789766831 (517h19m36.660849866s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789924742 (517h19m36.661007788s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51): protocol: read tcp 188.166.225.170:43134->145.79.227.43:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789933699 (517h19m36.661016734s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790128380 (517h19m36.661211425s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1): protocol: read tcp 188.166.225.170:46462->145.79.227.25:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.790137838 (517h19m36.661220873s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790156633 (517h19m36.661239679s) [seg_pos={0 0}, upload=453889]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:48:36.790241626 (517h19m36.661324661s) [upload=453889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:48:36.790246826 (517h19m36.661329861s) [upload=453889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:48:37.084198870 (517h19m36.955281915s) [upload=453889]: Done uploading segments... Dec 27 04:48:37.084219929 (517h19m36.955302974s) [upload=453889]: Upload finished. err: <nil> Duration: 42.028854191s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 3.499556463s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 04:56:45 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.499583856s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X....XXXX.X.X....X...XX.XXX..XXX..XXXX...XX...X.XX....XXXX..] duration: 6.282379694s trace: https://jaeger.dev.storj.io/trace/69f0122fd61b9ab6 FAIL (2025-12-27 04:56:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: YISXAFEDABFH3CL6I5K4VAVNLORSUTZQQQCQEPKEOB5P7C5YHPEA) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 5YRXH5Z2EJFGZQ5QDFJDS4KZOV7E6K2TJ2K5QA43KDUNYCG6DQDQ) context deadline exceeded ======LOGS====== Dec 27 04:54:53.228016161 (517h25m53.099099206s) [upload=453981]: Starting upload Dec 27 04:54:53.228085944 (517h25m53.099168989s) [upload=453981]: Done starting upload Dec 27 04:54:53.228222013 (517h25m53.099305058s) [upload=453981]: Uploading segments... Dec 27 04:54:53.228328165 (517h25m53.09941121s) [upload=453981]: Got next segment. Inline: false Dec 27 04:54:53.228364304 (517h25m53.099447349s) [upload=453981]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:54:53.228369984 (517h25m53.099453029s) [upload=453981]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:54:53.471176480 (517h25m53.342259515s) [seg_pos={0 0}, upload=453981]: Begin upload segment... Dec 27 04:54:53.471236604 (517h25m53.342319639s) [seg_pos={0 0}, upload=453981]: Done begin upload segment. Dec 27 04:54:53.471278964 (517h25m53.342361999s) [noise=true , address=145.79.227.10:4403 , piece_id=QJT2FXUJCWMPC5BEEU6INHKZEGCR2QGXLDW5YO55BPIIOU7REU3Q, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.471300586 (517h25m53.342383621s) [noise=true , address=77.90.31.15:4700 , piece_id=LRNJ473CM2CAZZVLZJA5LCSXWXMLGM5TBSB6Z6CHLYHU4UQDMKGA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.471799074 (517h25m53.34288211s) [noise=true , address=77.90.31.12:4590 , piece_id=W2JYHHHVVGT2D27ZYZA3HHKLO7Z5ATUAKJGR5Q5MPHVAZHGMAWWA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.472138040 (517h25m53.343221085s) [noise=true , address=145.79.227.2:4115 , piece_id=VHT5DS4Z5NJSNTNVGEERXQTAUKAUIHI7TOWPQBMZCN3QVFQQXSFQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.473034377 (517h25m53.344117412s) [noise=true , address=145.79.227.45:4229 , piece_id=ILHC3BM7TDMHUS4ACIDFSZHPIYW7XPXELKJWP5CGO5AEE2737DYQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.473402668 (517h25m53.344485703s) [noise=true , address=45.91.116.6:4974 , piece_id=KWE2OYH3WXX5JTHPXC6ZALPLKLZRMZHXJUZVI5RG3Q5DEQEUKHKQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.473761913 (517h25m53.344844948s) [noise=true , address=145.79.227.15:4874 , piece_id=R2MHZBZ63W47CYGDE3D7LPXSYLK4LTFJINUI3PJCBX3HKRIABK2A, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.474149651 (517h25m53.345232696s) [noise=true , address=45.91.116.5:4297 , piece_id=YISXAFEDABFH3CL6I5K4VAVNLORSUTZQQQCQEPKEOB5P7C5YHPEA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.475456570 (517h25m53.346539605s) [noise=true , address=77.90.31.21:4880 , piece_id=FVVU2G2JKJ6DTYHT62NAU7SM7M5C2NDERINEDXBTQH2PZTZAVVUQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.475590674 (517h25m53.346673719s) [noise=true , address=145.79.227.30:4762 , piece_id=TLFZPFKKCDWPPB7MRN5DAWIXCEIV6RIXTX7S3WOEHRRMRVTKJ66Q, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.476650784 (517h25m53.347733819s) [noise=true , address=77.90.31.11:4554 , piece_id=7A24GXKVWBS6KMNAQY5LEFQPS34FJVMUTSBWOCTAN4M24HHB3BCA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.477108996 (517h25m53.348192041s) [noise=true , address=145.79.227.11:4730 , piece_id=GAPDONNT4DIA5YW73JWWJKKNXLDC5UVIYD3IPMMV5OI3GRO3PBCA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.478711297 (517h25m53.349794332s) [noise=true , address=45.91.116.13:4154 , piece_id=5YRXH5Z2EJFGZQ5QDFJDS4KZOV7E6K2TJ2K5QA43KDUNYCG6DQDQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.480251329 (517h25m53.351334374s) [noise=true , address=145.79.227.23:4546 , piece_id=HWOYTQHNDDXXSET3DLLUKPB45NWJJWHRQ2RQQRY47E2HLEJRQBLA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.481777846 (517h25m53.352860881s) [noise=true , address=145.79.227.49:4337 , piece_id=H5D75YGDK6SBGCLYHPN2GTQXPFT46SLUPYNJB3AMCRDF2V5NC4KA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.483148365 (517h25m53.3542314s) [noise=true , address=145.79.227.35:4483 , piece_id=5GWKPQ2SPRO6JUS5AH5K3OJMLVK4U5UQMPDKXUN5NPYZXJT656EA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:55.192400484 (517h25m55.063483519s) [upload=453981]: Next returned nil segment Dec 27 04:54:55.192418167 (517h25m55.063501202s) [upload=453981]: Waiting for error group managing segments... Dec 27 04:54:55.477444407 (517h25m55.348527452s) [noise=true , address=145.79.227.15:4874 , piece_id=R2MHZBZ63W47CYGDE3D7LPXSYLK4LTFJINUI3PJCBX3HKRIABK2A, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.480670369 (517h25m55.351753414s) [noise=true , address=145.79.227.10:4403 , piece_id=QJT2FXUJCWMPC5BEEU6INHKZEGCR2QGXLDW5YO55BPIIOU7REU3Q, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.481618815 (517h25m55.35270187s) [noise=true , address=145.79.227.23:4546 , piece_id=HWOYTQHNDDXXSET3DLLUKPB45NWJJWHRQ2RQQRY47E2HLEJRQBLA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.494611344 (517h25m55.365694389s) [noise=true , address=145.79.227.2:4115 , piece_id=VHT5DS4Z5NJSNTNVGEERXQTAUKAUIHI7TOWPQBMZCN3QVFQQXSFQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.499504542 (517h25m55.370587577s) [noise=true , address=145.79.227.35:4483 , piece_id=5GWKPQ2SPRO6JUS5AH5K3OJMLVK4U5UQMPDKXUN5NPYZXJT656EA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.504255118 (517h25m55.375338153s) [noise=true , address=145.79.227.45:4229 , piece_id=ILHC3BM7TDMHUS4ACIDFSZHPIYW7XPXELKJWP5CGO5AEE2737DYQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.508654526 (517h25m55.379737561s) [noise=true , address=145.79.227.11:4730 , piece_id=GAPDONNT4DIA5YW73JWWJKKNXLDC5UVIYD3IPMMV5OI3GRO3PBCA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.509571982 (517h25m55.380655027s) [noise=true , address=145.79.227.49:4337 , piece_id=H5D75YGDK6SBGCLYHPN2GTQXPFT46SLUPYNJB3AMCRDF2V5NC4KA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.664817878 (517h25m55.535900913s) [noise=true , address=77.90.31.12:4590 , piece_id=W2JYHHHVVGT2D27ZYZA3HHKLO7Z5ATUAKJGR5Q5MPHVAZHGMAWWA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.674947937 (517h25m55.546030982s) [noise=true , address=77.90.31.11:4554 , piece_id=7A24GXKVWBS6KMNAQY5LEFQPS34FJVMUTSBWOCTAN4M24HHB3BCA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.774217743 (517h25m55.645300788s) [noise=true , address=77.90.31.21:4880 , piece_id=FVVU2G2JKJ6DTYHT62NAU7SM7M5C2NDERINEDXBTQH2PZTZAVVUQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.836321479 (517h25m55.707404524s) [noise=true , address=45.91.116.5:4297 , piece_id=YISXAFEDABFH3CL6I5K4VAVNLORSUTZQQQCQEPKEOB5P7C5YHPEA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:56.479032291 (517h25m56.350115336s) [noise=true , address=45.91.116.13:4154 , piece_id=5YRXH5Z2EJFGZQ5QDFJDS4KZOV7E6K2TJ2K5QA43KDUNYCG6DQDQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:56.479057088 (517h25m56.350140123s) [seg_pos={0 0}, upload=453981]: Segment reached optimal threshold of 13 pieces. Dec 27 04:54:56.479269112 (517h25m56.350352157s) [noise=true , address=145.79.227.30:4762 , piece_id=TLFZPFKKCDWPPB7MRN5DAWIXCEIV6RIXTX7S3WOEHRRMRVTKJ66Q, seg_pos={0 0}, upload=453981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 27 04:54:56.479278970 (517h25m56.350362005s) [noise=true , address=145.79.227.30:4762 , piece_id=TLFZPFKKCDWPPB7MRN5DAWIXCEIV6RIXTX7S3WOEHRRMRVTKJ66Q, seg_pos={0 0}, upload=453981]: Long tail cancellation due to optimal threshold Dec 27 04:54:56.479391665 (517h25m56.3504747s) [noise=true , address=77.90.31.15:4700 , piece_id=LRNJ473CM2CAZZVLZJA5LCSXWXMLGM5TBSB6Z6CHLYHU4UQDMKGA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 27 04:54:56.479407575 (517h25m56.35049061s) [noise=true , address=77.90.31.15:4700 , piece_id=LRNJ473CM2CAZZVLZJA5LCSXWXMLGM5TBSB6Z6CHLYHU4UQDMKGA, seg_pos={0 0}, upload=453981]: Long tail cancellation due to optimal threshold Dec 27 04:54:56.479491635 (517h25m56.35057467s) [noise=true , address=45.91.116.6:4974 , piece_id=KWE2OYH3WXX5JTHPXC6ZALPLKLZRMZHXJUZVI5RG3Q5DEQEUKHKQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 27 04:54:56.479501274 (517h25m56.350584319s) [noise=true , address=45.91.116.6:4974 , piece_id=KWE2OYH3WXX5JTHPXC6ZALPLKLZRMZHXJUZVI5RG3Q5DEQEUKHKQ, seg_pos={0 0}, upload=453981]: Long tail cancellation due to optimal threshold Dec 27 04:54:56.479548243 (517h25m56.350631278s) [seg_pos={0 0}, upload=453981]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:54:56.479628085 (517h25m56.35071113s) [upload=453981]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:54:56.479633856 (517h25m56.350716901s) [upload=453981]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:54:56.771755175 (517h25m56.642838211s) [upload=453981]: Done uploading segments... Dec 27 04:54:56.771779040 (517h25m56.642862075s) [upload=453981]: Upload finished. err: <nil> Duration: 6.282410013s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 3.731228407s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 04:56:03 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.731257423s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X......................X........] duration: 49.400434567s trace: https://jaeger.dev.storj.io/trace/60d70ba2fe18218 FAIL (2025-12-27 04:56:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 4 (517h19m18.685148379s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814531152 (517h19m18.685614187s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814852964 (517h19m18.685936009s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815170900 (517h19m18.686253945s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815486661 (517h19m18.686569706s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815835786 (517h19m18.686918831s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.816207564 (517h19m18.687290609s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818225116 (517h19m18.689308151s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818325547 (517h19m18.689408592s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.821132431 (517h19m18.692215466s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.822845684 (517h19m18.693928719s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.824299251 (517h19m18.695382296s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.825754102 (517h19m18.696837147s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.827122548 (517h19m18.698205593s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.828453653 (517h19m18.699536688s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:20.114191076 (517h19m19.985274121s) [upload=453889]: Next returned nil segment Dec 27 04:48:20.114206816 (517h19m19.985289851s) [upload=453889]: Waiting for error group managing segments... Dec 27 04:48:20.444209056 (517h19m20.315292102s) [noise=true , address=64.49.13.154:4929 , piece_id=2EXG7BEQ6NKAG3TNAH3TZ3K5ZI4KNOQYF3OEZMFWVX6BBMWNVXUQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.474597952 (517h19m20.345680987s) [noise=true , address=64.49.13.27:4964 , piece_id=XAROGTY2JLVFUVDYT7N7ZCABVYDMXA45NZF36NBAV4H72M3PJSCA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.478601025 (517h19m20.34968406s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.888413067 (517h19m20.759496102s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.930510167 (517h19m20.801593212s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.007760305 (517h19m20.87884334s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.041631362 (517h19m20.912714407s) [noise=true , address=77.90.31.14:4691 , piece_id=HIFLW7CWEYK6A6BZAYPDZLIUNA26IFXSFNVYV2UK4VPKEWYUMYJA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.052961556 (517h19m20.924044601s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:23.606703638 (517h19m23.477786673s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.046520345 (517h19m25.91760338s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.518771011 (517h19m26.389854046s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.589256778 (517h19m26.460339823s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:27.044280620 (517h19m26.915363655s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789125901 (517h19m36.660208936s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789145728 (517h19m36.660228764s) [seg_pos={0 0}, upload=453889]: Segment reached optimal threshold of 14 pieces. Dec 27 04:48:36.789615964 (517h19m36.660699009s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: read tcp 188.166.225.170:50246->145.79.227.40:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789624180 (517h19m36.660707215s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789761231 (517h19m36.660844276s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: read tcp 188.166.225.170:45976->145.79.227.44:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789766831 (517h19m36.660849866s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789924742 (517h19m36.661007788s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51): protocol: read tcp 188.166.225.170:43134->145.79.227.43:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789933699 (517h19m36.661016734s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790128380 (517h19m36.661211425s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1): protocol: read tcp 188.166.225.170:46462->145.79.227.25:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.790137838 (517h19m36.661220873s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790156633 (517h19m36.661239679s) [seg_pos={0 0}, upload=453889]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:48:36.790241626 (517h19m36.661324661s) [upload=453889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:48:36.790246826 (517h19m36.661329861s) [upload=453889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:48:37.084198870 (517h19m36.955281915s) [upload=453889]: Done uploading segments... Dec 27 04:48:37.084219929 (517h19m36.955302974s) [upload=453889]: Upload finished. err: <nil> Duration: 49.40045755s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.X....XXXX.X.X....X...XX.XXX..XXX..XXXX...XX...X.XX....XXXX.] duration: 22.109102369s trace: https://jaeger.dev.storj.io/trace/69f0122fd61b9ab6 FAIL (2025-12-27 04:55:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: YISXAFEDABFH3CL6I5K4VAVNLORSUTZQQQCQEPKEOB5P7C5YHPEA) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 5YRXH5Z2EJFGZQ5QDFJDS4KZOV7E6K2TJ2K5QA43KDUNYCG6DQDQ) context deadline exceeded ======LOGS====== Dec 27 04:54:53.228016161 (517h25m53.099099206s) [upload=453981]: Starting upload Dec 27 04:54:53.228085944 (517h25m53.099168989s) [upload=453981]: Done starting upload Dec 27 04:54:53.228222013 (517h25m53.099305058s) [upload=453981]: Uploading segments... Dec 27 04:54:53.228328165 (517h25m53.09941121s) [upload=453981]: Got next segment. Inline: false Dec 27 04:54:53.228364304 (517h25m53.099447349s) [upload=453981]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:54:53.228369984 (517h25m53.099453029s) [upload=453981]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:54:53.471176480 (517h25m53.342259515s) [seg_pos={0 0}, upload=453981]: Begin upload segment... Dec 27 04:54:53.471236604 (517h25m53.342319639s) [seg_pos={0 0}, upload=453981]: Done begin upload segment. Dec 27 04:54:53.471278964 (517h25m53.342361999s) [noise=true , address=145.79.227.10:4403 , piece_id=QJT2FXUJCWMPC5BEEU6INHKZEGCR2QGXLDW5YO55BPIIOU7REU3Q, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.471300586 (517h25m53.342383621s) [noise=true , address=77.90.31.15:4700 , piece_id=LRNJ473CM2CAZZVLZJA5LCSXWXMLGM5TBSB6Z6CHLYHU4UQDMKGA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.471799074 (517h25m53.34288211s) [noise=true , address=77.90.31.12:4590 , piece_id=W2JYHHHVVGT2D27ZYZA3HHKLO7Z5ATUAKJGR5Q5MPHVAZHGMAWWA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.472138040 (517h25m53.343221085s) [noise=true , address=145.79.227.2:4115 , piece_id=VHT5DS4Z5NJSNTNVGEERXQTAUKAUIHI7TOWPQBMZCN3QVFQQXSFQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.473034377 (517h25m53.344117412s) [noise=true , address=145.79.227.45:4229 , piece_id=ILHC3BM7TDMHUS4ACIDFSZHPIYW7XPXELKJWP5CGO5AEE2737DYQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.473402668 (517h25m53.344485703s) [noise=true , address=45.91.116.6:4974 , piece_id=KWE2OYH3WXX5JTHPXC6ZALPLKLZRMZHXJUZVI5RG3Q5DEQEUKHKQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.473761913 (517h25m53.344844948s) [noise=true , address=145.79.227.15:4874 , piece_id=R2MHZBZ63W47CYGDE3D7LPXSYLK4LTFJINUI3PJCBX3HKRIABK2A, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.474149651 (517h25m53.345232696s) [noise=true , address=45.91.116.5:4297 , piece_id=YISXAFEDABFH3CL6I5K4VAVNLORSUTZQQQCQEPKEOB5P7C5YHPEA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.475456570 (517h25m53.346539605s) [noise=true , address=77.90.31.21:4880 , piece_id=FVVU2G2JKJ6DTYHT62NAU7SM7M5C2NDERINEDXBTQH2PZTZAVVUQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.475590674 (517h25m53.346673719s) [noise=true , address=145.79.227.30:4762 , piece_id=TLFZPFKKCDWPPB7MRN5DAWIXCEIV6RIXTX7S3WOEHRRMRVTKJ66Q, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.476650784 (517h25m53.347733819s) [noise=true , address=77.90.31.11:4554 , piece_id=7A24GXKVWBS6KMNAQY5LEFQPS34FJVMUTSBWOCTAN4M24HHB3BCA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.477108996 (517h25m53.348192041s) [noise=true , address=145.79.227.11:4730 , piece_id=GAPDONNT4DIA5YW73JWWJKKNXLDC5UVIYD3IPMMV5OI3GRO3PBCA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.478711297 (517h25m53.349794332s) [noise=true , address=45.91.116.13:4154 , piece_id=5YRXH5Z2EJFGZQ5QDFJDS4KZOV7E6K2TJ2K5QA43KDUNYCG6DQDQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.480251329 (517h25m53.351334374s) [noise=true , address=145.79.227.23:4546 , piece_id=HWOYTQHNDDXXSET3DLLUKPB45NWJJWHRQ2RQQRY47E2HLEJRQBLA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.481777846 (517h25m53.352860881s) [noise=true , address=145.79.227.49:4337 , piece_id=H5D75YGDK6SBGCLYHPN2GTQXPFT46SLUPYNJB3AMCRDF2V5NC4KA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.483148365 (517h25m53.3542314s) [noise=true , address=145.79.227.35:4483 , piece_id=5GWKPQ2SPRO6JUS5AH5K3OJMLVK4U5UQMPDKXUN5NPYZXJT656EA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:55.192400484 (517h25m55.063483519s) [upload=453981]: Next returned nil segment Dec 27 04:54:55.192418167 (517h25m55.063501202s) [upload=453981]: Waiting for error group managing segments... Dec 27 04:54:55.477444407 (517h25m55.348527452s) [noise=true , address=145.79.227.15:4874 , piece_id=R2MHZBZ63W47CYGDE3D7LPXSYLK4LTFJINUI3PJCBX3HKRIABK2A, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.480670369 (517h25m55.351753414s) [noise=true , address=145.79.227.10:4403 , piece_id=QJT2FXUJCWMPC5BEEU6INHKZEGCR2QGXLDW5YO55BPIIOU7REU3Q, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.481618815 (517h25m55.35270187s) [noise=true , address=145.79.227.23:4546 , piece_id=HWOYTQHNDDXXSET3DLLUKPB45NWJJWHRQ2RQQRY47E2HLEJRQBLA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.494611344 (517h25m55.365694389s) [noise=true , address=145.79.227.2:4115 , piece_id=VHT5DS4Z5NJSNTNVGEERXQTAUKAUIHI7TOWPQBMZCN3QVFQQXSFQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.499504542 (517h25m55.370587577s) [noise=true , address=145.79.227.35:4483 , piece_id=5GWKPQ2SPRO6JUS5AH5K3OJMLVK4U5UQMPDKXUN5NPYZXJT656EA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.504255118 (517h25m55.375338153s) [noise=true , address=145.79.227.45:4229 , piece_id=ILHC3BM7TDMHUS4ACIDFSZHPIYW7XPXELKJWP5CGO5AEE2737DYQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.508654526 (517h25m55.379737561s) [noise=true , address=145.79.227.11:4730 , piece_id=GAPDONNT4DIA5YW73JWWJKKNXLDC5UVIYD3IPMMV5OI3GRO3PBCA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.509571982 (517h25m55.380655027s) [noise=true , address=145.79.227.49:4337 , piece_id=H5D75YGDK6SBGCLYHPN2GTQXPFT46SLUPYNJB3AMCRDF2V5NC4KA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.664817878 (517h25m55.535900913s) [noise=true , address=77.90.31.12:4590 , piece_id=W2JYHHHVVGT2D27ZYZA3HHKLO7Z5ATUAKJGR5Q5MPHVAZHGMAWWA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.674947937 (517h25m55.546030982s) [noise=true , address=77.90.31.11:4554 , piece_id=7A24GXKVWBS6KMNAQY5LEFQPS34FJVMUTSBWOCTAN4M24HHB3BCA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.774217743 (517h25m55.645300788s) [noise=true , address=77.90.31.21:4880 , piece_id=FVVU2G2JKJ6DTYHT62NAU7SM7M5C2NDERINEDXBTQH2PZTZAVVUQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.836321479 (517h25m55.707404524s) [noise=true , address=45.91.116.5:4297 , piece_id=YISXAFEDABFH3CL6I5K4VAVNLORSUTZQQQCQEPKEOB5P7C5YHPEA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:56.479032291 (517h25m56.350115336s) [noise=true , address=45.91.116.13:4154 , piece_id=5YRXH5Z2EJFGZQ5QDFJDS4KZOV7E6K2TJ2K5QA43KDUNYCG6DQDQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:56.479057088 (517h25m56.350140123s) [seg_pos={0 0}, upload=453981]: Segment reached optimal threshold of 13 pieces. Dec 27 04:54:56.479269112 (517h25m56.350352157s) [noise=true , address=145.79.227.30:4762 , piece_id=TLFZPFKKCDWPPB7MRN5DAWIXCEIV6RIXTX7S3WOEHRRMRVTKJ66Q, seg_pos={0 0}, upload=453981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 27 04:54:56.479278970 (517h25m56.350362005s) [noise=true , address=145.79.227.30:4762 , piece_id=TLFZPFKKCDWPPB7MRN5DAWIXCEIV6RIXTX7S3WOEHRRMRVTKJ66Q, seg_pos={0 0}, upload=453981]: Long tail cancellation due to optimal threshold Dec 27 04:54:56.479391665 (517h25m56.3504747s) [noise=true , address=77.90.31.15:4700 , piece_id=LRNJ473CM2CAZZVLZJA5LCSXWXMLGM5TBSB6Z6CHLYHU4UQDMKGA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 27 04:54:56.479407575 (517h25m56.35049061s) [noise=true , address=77.90.31.15:4700 , piece_id=LRNJ473CM2CAZZVLZJA5LCSXWXMLGM5TBSB6Z6CHLYHU4UQDMKGA, seg_pos={0 0}, upload=453981]: Long tail cancellation due to optimal threshold Dec 27 04:54:56.479491635 (517h25m56.35057467s) [noise=true , address=45.91.116.6:4974 , piece_id=KWE2OYH3WXX5JTHPXC6ZALPLKLZRMZHXJUZVI5RG3Q5DEQEUKHKQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 27 04:54:56.479501274 (517h25m56.350584319s) [noise=true , address=45.91.116.6:4974 , piece_id=KWE2OYH3WXX5JTHPXC6ZALPLKLZRMZHXJUZVI5RG3Q5DEQEUKHKQ, seg_pos={0 0}, upload=453981]: Long tail cancellation due to optimal threshold Dec 27 04:54:56.479548243 (517h25m56.350631278s) [seg_pos={0 0}, upload=453981]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:54:56.479628085 (517h25m56.35071113s) [upload=453981]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:54:56.479633856 (517h25m56.350716901s) [upload=453981]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:54:56.771755175 (517h25m56.642838211s) [upload=453981]: Done uploading segments... Dec 27 04:54:56.771779040 (517h25m56.642862075s) [upload=453981]: Upload finished. err: <nil> Duration: 22.109120533s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 3.505086775s trace: https://jaeger.dev.storj.io/trace/529442d366991f4e FAIL (2025-12-27 04:55:31 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.505112604s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [..X....XXXX.X.X....X...XX.XXX..XXX..XXXX...XX...X.XX....XXXX] duration: 30.002919794s trace: https://jaeger.dev.storj.io/trace/69f0122fd61b9ab6 FAIL (2025-12-27 04:55:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: YISXAFEDABFH3CL6I5K4VAVNLORSUTZQQQCQEPKEOB5P7C5YHPEA) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 5YRXH5Z2EJFGZQ5QDFJDS4KZOV7E6K2TJ2K5QA43KDUNYCG6DQDQ) context deadline exceeded ======LOGS====== Dec 27 04:54:53.228016161 (517h25m53.099099206s) [upload=453981]: Starting upload Dec 27 04:54:53.228085944 (517h25m53.099168989s) [upload=453981]: Done starting upload Dec 27 04:54:53.228222013 (517h25m53.099305058s) [upload=453981]: Uploading segments... Dec 27 04:54:53.228328165 (517h25m53.09941121s) [upload=453981]: Got next segment. Inline: false Dec 27 04:54:53.228364304 (517h25m53.099447349s) [upload=453981]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 04:54:53.228369984 (517h25m53.099453029s) [upload=453981]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 04:54:53.471176480 (517h25m53.342259515s) [seg_pos={0 0}, upload=453981]: Begin upload segment... Dec 27 04:54:53.471236604 (517h25m53.342319639s) [seg_pos={0 0}, upload=453981]: Done begin upload segment. Dec 27 04:54:53.471278964 (517h25m53.342361999s) [noise=true , address=145.79.227.10:4403 , piece_id=QJT2FXUJCWMPC5BEEU6INHKZEGCR2QGXLDW5YO55BPIIOU7REU3Q, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.471300586 (517h25m53.342383621s) [noise=true , address=77.90.31.15:4700 , piece_id=LRNJ473CM2CAZZVLZJA5LCSXWXMLGM5TBSB6Z6CHLYHU4UQDMKGA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.471799074 (517h25m53.34288211s) [noise=true , address=77.90.31.12:4590 , piece_id=W2JYHHHVVGT2D27ZYZA3HHKLO7Z5ATUAKJGR5Q5MPHVAZHGMAWWA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.472138040 (517h25m53.343221085s) [noise=true , address=145.79.227.2:4115 , piece_id=VHT5DS4Z5NJSNTNVGEERXQTAUKAUIHI7TOWPQBMZCN3QVFQQXSFQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.473034377 (517h25m53.344117412s) [noise=true , address=145.79.227.45:4229 , piece_id=ILHC3BM7TDMHUS4ACIDFSZHPIYW7XPXELKJWP5CGO5AEE2737DYQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.473402668 (517h25m53.344485703s) [noise=true , address=45.91.116.6:4974 , piece_id=KWE2OYH3WXX5JTHPXC6ZALPLKLZRMZHXJUZVI5RG3Q5DEQEUKHKQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.473761913 (517h25m53.344844948s) [noise=true , address=145.79.227.15:4874 , piece_id=R2MHZBZ63W47CYGDE3D7LPXSYLK4LTFJINUI3PJCBX3HKRIABK2A, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.474149651 (517h25m53.345232696s) [noise=true , address=45.91.116.5:4297 , piece_id=YISXAFEDABFH3CL6I5K4VAVNLORSUTZQQQCQEPKEOB5P7C5YHPEA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.475456570 (517h25m53.346539605s) [noise=true , address=77.90.31.21:4880 , piece_id=FVVU2G2JKJ6DTYHT62NAU7SM7M5C2NDERINEDXBTQH2PZTZAVVUQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.475590674 (517h25m53.346673719s) [noise=true , address=145.79.227.30:4762 , piece_id=TLFZPFKKCDWPPB7MRN5DAWIXCEIV6RIXTX7S3WOEHRRMRVTKJ66Q, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.476650784 (517h25m53.347733819s) [noise=true , address=77.90.31.11:4554 , piece_id=7A24GXKVWBS6KMNAQY5LEFQPS34FJVMUTSBWOCTAN4M24HHB3BCA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.477108996 (517h25m53.348192041s) [noise=true , address=145.79.227.11:4730 , piece_id=GAPDONNT4DIA5YW73JWWJKKNXLDC5UVIYD3IPMMV5OI3GRO3PBCA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.478711297 (517h25m53.349794332s) [noise=true , address=45.91.116.13:4154 , piece_id=5YRXH5Z2EJFGZQ5QDFJDS4KZOV7E6K2TJ2K5QA43KDUNYCG6DQDQ, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.480251329 (517h25m53.351334374s) [noise=true , address=145.79.227.23:4546 , piece_id=HWOYTQHNDDXXSET3DLLUKPB45NWJJWHRQ2RQQRY47E2HLEJRQBLA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.481777846 (517h25m53.352860881s) [noise=true , address=145.79.227.49:4337 , piece_id=H5D75YGDK6SBGCLYHPN2GTQXPFT46SLUPYNJB3AMCRDF2V5NC4KA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:53.483148365 (517h25m53.3542314s) [noise=true , address=145.79.227.35:4483 , piece_id=5GWKPQ2SPRO6JUS5AH5K3OJMLVK4U5UQMPDKXUN5NPYZXJT656EA, seg_pos={0 0}, upload=453981]: Uploading piece... Dec 27 04:54:55.192400484 (517h25m55.063483519s) [upload=453981]: Next returned nil segment Dec 27 04:54:55.192418167 (517h25m55.063501202s) [upload=453981]: Waiting for error group managing segments... Dec 27 04:54:55.477444407 (517h25m55.348527452s) [noise=true , address=145.79.227.15:4874 , piece_id=R2MHZBZ63W47CYGDE3D7LPXSYLK4LTFJINUI3PJCBX3HKRIABK2A, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.480670369 (517h25m55.351753414s) [noise=true , address=145.79.227.10:4403 , piece_id=QJT2FXUJCWMPC5BEEU6INHKZEGCR2QGXLDW5YO55BPIIOU7REU3Q, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.481618815 (517h25m55.35270187s) [noise=true , address=145.79.227.23:4546 , piece_id=HWOYTQHNDDXXSET3DLLUKPB45NWJJWHRQ2RQQRY47E2HLEJRQBLA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.494611344 (517h25m55.365694389s) [noise=true , address=145.79.227.2:4115 , piece_id=VHT5DS4Z5NJSNTNVGEERXQTAUKAUIHI7TOWPQBMZCN3QVFQQXSFQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.499504542 (517h25m55.370587577s) [noise=true , address=145.79.227.35:4483 , piece_id=5GWKPQ2SPRO6JUS5AH5K3OJMLVK4U5UQMPDKXUN5NPYZXJT656EA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.504255118 (517h25m55.375338153s) [noise=true , address=145.79.227.45:4229 , piece_id=ILHC3BM7TDMHUS4ACIDFSZHPIYW7XPXELKJWP5CGO5AEE2737DYQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.508654526 (517h25m55.379737561s) [noise=true , address=145.79.227.11:4730 , piece_id=GAPDONNT4DIA5YW73JWWJKKNXLDC5UVIYD3IPMMV5OI3GRO3PBCA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.509571982 (517h25m55.380655027s) [noise=true , address=145.79.227.49:4337 , piece_id=H5D75YGDK6SBGCLYHPN2GTQXPFT46SLUPYNJB3AMCRDF2V5NC4KA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.664817878 (517h25m55.535900913s) [noise=true , address=77.90.31.12:4590 , piece_id=W2JYHHHVVGT2D27ZYZA3HHKLO7Z5ATUAKJGR5Q5MPHVAZHGMAWWA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.674947937 (517h25m55.546030982s) [noise=true , address=77.90.31.11:4554 , piece_id=7A24GXKVWBS6KMNAQY5LEFQPS34FJVMUTSBWOCTAN4M24HHB3BCA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.774217743 (517h25m55.645300788s) [noise=true , address=77.90.31.21:4880 , piece_id=FVVU2G2JKJ6DTYHT62NAU7SM7M5C2NDERINEDXBTQH2PZTZAVVUQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:55.836321479 (517h25m55.707404524s) [noise=true , address=45.91.116.5:4297 , piece_id=YISXAFEDABFH3CL6I5K4VAVNLORSUTZQQQCQEPKEOB5P7C5YHPEA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:56.479032291 (517h25m56.350115336s) [noise=true , address=45.91.116.13:4154 , piece_id=5YRXH5Z2EJFGZQ5QDFJDS4KZOV7E6K2TJ2K5QA43KDUNYCG6DQDQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: <nil> Dec 27 04:54:56.479057088 (517h25m56.350140123s) [seg_pos={0 0}, upload=453981]: Segment reached optimal threshold of 13 pieces. Dec 27 04:54:56.479269112 (517h25m56.350352157s) [noise=true , address=145.79.227.30:4762 , piece_id=TLFZPFKKCDWPPB7MRN5DAWIXCEIV6RIXTX7S3WOEHRRMRVTKJ66Q, seg_pos={0 0}, upload=453981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 27 04:54:56.479278970 (517h25m56.350362005s) [noise=true , address=145.79.227.30:4762 , piece_id=TLFZPFKKCDWPPB7MRN5DAWIXCEIV6RIXTX7S3WOEHRRMRVTKJ66Q, seg_pos={0 0}, upload=453981]: Long tail cancellation due to optimal threshold Dec 27 04:54:56.479391665 (517h25m56.3504747s) [noise=true , address=77.90.31.15:4700 , piece_id=LRNJ473CM2CAZZVLZJA5LCSXWXMLGM5TBSB6Z6CHLYHU4UQDMKGA, seg_pos={0 0}, upload=453981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 27 04:54:56.479407575 (517h25m56.35049061s) [noise=true , address=77.90.31.15:4700 , piece_id=LRNJ473CM2CAZZVLZJA5LCSXWXMLGM5TBSB6Z6CHLYHU4UQDMKGA, seg_pos={0 0}, upload=453981]: Long tail cancellation due to optimal threshold Dec 27 04:54:56.479491635 (517h25m56.35057467s) [noise=true , address=45.91.116.6:4974 , piece_id=KWE2OYH3WXX5JTHPXC6ZALPLKLZRMZHXJUZVI5RG3Q5DEQEUKHKQ, seg_pos={0 0}, upload=453981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 27 04:54:56.479501274 (517h25m56.350584319s) [noise=true , address=45.91.116.6:4974 , piece_id=KWE2OYH3WXX5JTHPXC6ZALPLKLZRMZHXJUZVI5RG3Q5DEQEUKHKQ, seg_pos={0 0}, upload=453981]: Long tail cancellation due to optimal threshold Dec 27 04:54:56.479548243 (517h25m56.350631278s) [seg_pos={0 0}, upload=453981]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 04:54:56.479628085 (517h25m56.35071113s) [upload=453981]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:54:56.479633856 (517h25m56.350716901s) [upload=453981]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:54:56.771755175 (517h25m56.642838211s) [upload=453981]: Done uploading segments... Dec 27 04:54:56.771779040 (517h25m56.642862075s) [upload=453981]: Upload finished. err: <nil> Duration: 30.002982605s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X......................X.......] duration: 21.525440383s trace: https://jaeger.dev.storj.io/trace/60d70ba2fe18218 FAIL (2025-12-27 04:54:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 4 (517h19m18.685148379s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814531152 (517h19m18.685614187s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.814852964 (517h19m18.685936009s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815170900 (517h19m18.686253945s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815486661 (517h19m18.686569706s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.815835786 (517h19m18.686918831s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.816207564 (517h19m18.687290609s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818225116 (517h19m18.689308151s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.818325547 (517h19m18.689408592s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.821132431 (517h19m18.692215466s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.822845684 (517h19m18.693928719s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.824299251 (517h19m18.695382296s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.825754102 (517h19m18.696837147s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.827122548 (517h19m18.698205593s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:18.828453653 (517h19m18.699536688s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Uploading piece... Dec 27 04:48:20.114191076 (517h19m19.985274121s) [upload=453889]: Next returned nil segment Dec 27 04:48:20.114206816 (517h19m19.985289851s) [upload=453889]: Waiting for error group managing segments... Dec 27 04:48:20.444209056 (517h19m20.315292102s) [noise=true , address=64.49.13.154:4929 , piece_id=2EXG7BEQ6NKAG3TNAH3TZ3K5ZI4KNOQYF3OEZMFWVX6BBMWNVXUQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.474597952 (517h19m20.345680987s) [noise=true , address=64.49.13.27:4964 , piece_id=XAROGTY2JLVFUVDYT7N7ZCABVYDMXA45NZF36NBAV4H72M3PJSCA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.478601025 (517h19m20.34968406s) [noise=true , address=64.49.13.139:4473 , piece_id=24KFUZYUQKJ3CGQSXFTTD3U4T7SFHTJDETCYO5EDH2ZFPN2KKQSQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.888413067 (517h19m20.759496102s) [noise=true , address=77.90.31.10:4538 , piece_id=NWZ2VRBM32PP63XO25NN6X5S7LNHZIBJGI7CIWKHYTGQAWW5KH3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:20.930510167 (517h19m20.801593212s) [noise=true , address=45.91.116.13:4171 , piece_id=CRMKVXXPJNVJKXITDT43FRSIJ5QZI2PJ7WFLK7E7BD6D6NHCAU5A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.007760305 (517h19m20.87884334s) [noise=true , address=77.90.31.21:4911 , piece_id=6RC5Z7BXDCFDRPLQO3PVMB3X53X4MUNEGLVZWRJEE4V62F4V47KQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.041631362 (517h19m20.912714407s) [noise=true , address=77.90.31.14:4691 , piece_id=HIFLW7CWEYK6A6BZAYPDZLIUNA26IFXSFNVYV2UK4VPKEWYUMYJA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:21.052961556 (517h19m20.924044601s) [noise=true , address=77.90.31.18:4830 , piece_id=LPBQKODNMNSFCPZTA5F37QGSWEVZUYKMMFGIQDMGPKAN5EYQDVPQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:23.606703638 (517h19m23.477786673s) [noise=true , address=145.79.227.33:4888 , piece_id=7SLZH3CHVOU4AAZKXIZRY44KCI7B757JIRIDJIKUSI63HBIIBKMQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.046520345 (517h19m25.91760338s) [noise=true , address=145.79.227.29:4962 , piece_id=EFE27B5QVL3FD7AHC4SUHTOTDPTZ5XPFOGFF3RQXOC6RS6B2674A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.518771011 (517h19m26.389854046s) [noise=true , address=77.90.31.13:4660 , piece_id=XE3YRDPLM5FNADIGGO5LAHAENOEMJY4XYQTSAYVTXHZM5FC6FSNQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:26.589256778 (517h19m26.460339823s) [noise=true , address=145.79.227.30:4791 , piece_id=CEMBFXMBANYHYP4HPIFWR7DRLOGGDYROI5ITYK76LGSKXRQEYS3A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:27.044280620 (517h19m26.915363655s) [noise=true , address=145.79.227.34:4932 , piece_id=FBA3EOFHKRLVEYF3AN3V5PQAPCB3GDNHZTYHIWLQ4YIFLPWWJO7A, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789125901 (517h19m36.660208936s) [noise=true , address=64.49.13.169:4706 , piece_id=LM3Z4DD5DFNGFWHHZQL3AL7YCX55RH5BIVG5GXKGFADM333VNEVQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: <nil> Dec 27 04:48:36.789145728 (517h19m36.660228764s) [seg_pos={0 0}, upload=453889]: Segment reached optimal threshold of 14 pieces. Dec 27 04:48:36.789615964 (517h19m36.660699009s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: read tcp 188.166.225.170:50246->145.79.227.40:4069: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789624180 (517h19m36.660707215s) [noise=true , address=145.79.227.40:4069 , piece_id=4TKMGKUUFOEFS5SHCGNES2JKQZHONUKPHH6Z7S7DDX7E2XOA6ZQA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789761231 (517h19m36.660844276s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6): protocol: read tcp 188.166.225.170:45976->145.79.227.44:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789766831 (517h19m36.660849866s) [noise=true , address=145.79.227.44:4223 , piece_id=GXVY4FLH67OF24IOGKCWW7GKJTPLVRT4GITCI6S33OOJK3XGVNDA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.789924742 (517h19m36.661007788s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51): protocol: read tcp 188.166.225.170:43134->145.79.227.43:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.789933699 (517h19m36.661016734s) [noise=true , address=145.79.227.43:4174 , piece_id=TCIZA7DZWHI726ZQM6K6YVXIT2H2DLP33CH7OGMTHZ5YCAJYOORA, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790128380 (517h19m36.661211425s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1): protocol: read tcp 188.166.225.170:46462->145.79.227.25:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 04:48:36.790137838 (517h19m36.661220873s) [noise=true , address=145.79.227.25:4624 , piece_id=UY7L3LJSWMT6GFBGX5Q2PUO7HMAGV3YI74G7GPBMQA5VS2EEY3RQ, seg_pos={0 0}, upload=453889]: Long tail cancellation due to optimal threshold Dec 27 04:48:36.790156633 (517h19m36.661239679s) [seg_pos={0 0}, upload=453889]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 04:48:36.790241626 (517h19m36.661324661s) [upload=453889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 04:48:36.790246826 (517h19m36.661329861s) [upload=453889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 04:48:37.084198870 (517h19m36.955281915s) [upload=453889]: Done uploading segments... Dec 27 04:48:37.084219929 (517h19m36.955302974s) [upload=453889]: Upload finished. err: <nil> Duration: 21.525468176s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)