Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.878783011s FAIL (2025-10-07 07:49:38 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== 2.684228524 (330h16m48.146321007s) [noise=true , address=145.79.227.22:4512 , piece_id=ZHYRXIBGXVLWMXK3JSTX5BM3KK3UYASIGFEA3HIKQEXWRPOXZKDA, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.684385934 (330h16m48.146478417s) [noise=true , address=45.91.116.14:4211 , piece_id=E6CAX7G3RVE22AJT7HOESRF633GR7YDDAHJLHOSYKXTMLUIFRGZA, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.684963844 (330h16m48.147056327s) [noise=true , address=77.90.31.14:4689 , piece_id=IVQS3GZNMI7JZFYVW3RHL4JSDVZ5UIHDNX5JEFV3346M3ZHQWNIQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.685221956 (330h16m48.147314439s) [noise=true , address=45.91.116.6:4995 , piece_id=3IJGK3G2ZYWWRMDWOVXCNCEMAWP5YQXPNIBP5BM2LPV6GVL2H6NQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.685452474 (330h16m48.147544957s) [noise=true , address=45.91.116.5:4302 , piece_id=DDYTKB75NQGJGPKPWC6AZ4RKDU6DBLA3AX2GNKEUUWJ227IDU5CQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.685900778 (330h16m48.147993261s) [noise=true , address=45.91.116.9:4272 , piece_id=EPDK76GOYFEJOFB4GJEB4WOKQ55L3PXKEJUQC66AUHE5BV3K3LBQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.685950814 (330h16m48.148043297s) [noise=true , address=45.91.116.12:4235 , piece_id=C3ZEP2EDRTNVHWMIH4OXWZVSACJRO3JO4FYMM5VTMGUWBFRSLRUA, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.686310157 (330h16m48.148402631s) [noise=true , address=77.90.31.10:4526 , piece_id=IZTN2WX2Q5P2ML4EZ2OGW23HQZ3UIXBQYWLUUTXJDFU73ZUELBMQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.686461365 (330h16m48.148553848s) [noise=true , address=77.90.31.16:4756 , piece_id=4WCDI36U4YAC3C6KLZUHS44YRYIJ6BNDCO5VPP35JYKD4WN6XQAQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.687006704 (330h16m48.149099187s) [noise=true , address=145.79.227.33:4900 , piece_id=5L3XYQELQU7KV7W466L5DCNXSP6U2LX7EIFOH6TUH67INZRGSWJQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.687022113 (330h16m48.149114596s) [noise=true , address=77.90.31.14:4684 , piece_id=6VCBHVVFPETZFSFBXLBCTVPLXL3AZXSFRNOFWJTCYNW75C4F2WBA, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.687498480 (330h16m48.149590963s) [noise=true , address=145.79.227.30:4777 , piece_id=UX4R7UC5EEX67KDL7SVKFPWD4VVB4WJPKAIBSRY5Y4IMAKRVKXYA, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.687559176 (330h16m48.149651699s) [noise=true , address=77.90.31.19:4863 , piece_id=2NVNF4CV7NQVKZQZ3QOUPSCNPETR4ZKHJRBVW6TJ4RAHMZBXH5CQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.688272183 (330h16m48.150364666s) [noise=true , address=77.90.31.21:4911 , piece_id=NOFTEDILTPXQ6LHAUDHP5GKZWI6J7VXGVJZNXYPZTUKQEZ5JO4OQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.690143407 (330h16m48.15223589s) [noise=true , address=145.79.227.29:4944 , piece_id=6IWBVDICAIROYEKIUBJZTSIZ6XI4KJHKY24F2L7BKHWGBUQA24LA, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.690606197 (330h16m48.152698671s) [noise=true , address=77.90.31.13:4643 , piece_id=UTPWYEFHE2B4BTPV7ANSJ7KUHYM4HF6NKRNPXX5NIAZQVRR3XDZQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.690968368 (330h16m48.153060851s) [noise=true , address=145.79.227.25:4639 , piece_id=SN5IMBBBZYXY4L24XQYZING6O2U7SBLFTPRWRU7EYAOGIGYG64IQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:04.184255931 (330h16m49.646348424s) [upload=290665]: Next returned nil segment Oct 7 07:48:04.184283844 (330h16m49.646376327s) [upload=290665]: Waiting for error group managing segments... Oct 7 07:48:04.558544184 (330h16m50.020636667s) [noise=true , address=145.79.227.22:4512 , piece_id=ZHYRXIBGXVLWMXK3JSTX5BM3KK3UYASIGFEA3HIKQEXWRPOXZKDA, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.559842245 (330h16m50.021934739s) [noise=true , address=145.79.227.30:4777 , piece_id=UX4R7UC5EEX67KDL7SVKFPWD4VVB4WJPKAIBSRY5Y4IMAKRVKXYA, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.567228070 (330h16m50.029320553s) [noise=true , address=145.79.227.23:4561 , piece_id=2UOAAEEW4AOLGFHNZTTFAHM3KJZTSDBO3MFCV2OMCN6TSJO46GZA, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.571144567 (330h16m50.03323706s) [noise=true , address=145.79.227.25:4639 , piece_id=SN5IMBBBZYXY4L24XQYZING6O2U7SBLFTPRWRU7EYAOGIGYG64IQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.571284323 (330h16m50.033376817s) [noise=true , address=145.79.227.29:4944 , piece_id=6IWBVDICAIROYEKIUBJZTSIZ6XI4KJHKY24F2L7BKHWGBUQA24LA, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.571824523 (330h16m50.033916996s) [noise=true , address=145.79.227.33:4900 , piece_id=5L3XYQELQU7KV7W466L5DCNXSP6U2LX7EIFOH6TUH67INZRGSWJQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.642083669 (330h16m50.104176162s) [noise=true , address=45.91.116.9:4272 , piece_id=EPDK76GOYFEJOFB4GJEB4WOKQ55L3PXKEJUQC66AUHE5BV3K3LBQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.652430721 (330h16m50.114523204s) [noise=true , address=45.91.116.12:4235 , piece_id=C3ZEP2EDRTNVHWMIH4OXWZVSACJRO3JO4FYMM5VTMGUWBFRSLRUA, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.660824113 (330h16m50.122916597s) [noise=true , address=77.90.31.21:4911 , piece_id=NOFTEDILTPXQ6LHAUDHP5GKZWI6J7VXGVJZNXYPZTUKQEZ5JO4OQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.680019976 (330h16m50.142112459s) [noise=true , address=45.91.116.14:4211 , piece_id=E6CAX7G3RVE22AJT7HOESRF633GR7YDDAHJLHOSYKXTMLUIFRGZA, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.705995618 (330h16m50.168088101s) [noise=true , address=77.90.31.14:4689 , piece_id=IVQS3GZNMI7JZFYVW3RHL4JSDVZ5UIHDNX5JEFV3346M3ZHQWNIQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.717852216 (330h16m50.179944699s) [noise=true , address=77.90.31.13:4643 , piece_id=UTPWYEFHE2B4BTPV7ANSJ7KUHYM4HF6NKRNPXX5NIAZQVRR3XDZQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.718777087 (330h16m50.18086957s) [noise=true , address=77.90.31.10:4526 , piece_id=IZTN2WX2Q5P2ML4EZ2OGW23HQZ3UIXBQYWLUUTXJDFU73ZUELBMQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.733559335 (330h16m50.195651819s) [noise=true , address=45.91.116.5:4302 , piece_id=DDYTKB75NQGJGPKPWC6AZ4RKDU6DBLA3AX2GNKEUUWJ227IDU5CQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.733600604 (330h16m50.195693087s) [seg_pos={0 0}, upload=290665]: Segment reached optimal threshold of 14 pieces. Oct 7 07:48:04.734223801 (330h16m50.196316294s) [noise=true , address=77.90.31.16:4756 , piece_id=4WCDI36U4YAC3C6KLZUHS44YRYIJ6BNDCO5VPP35JYKD4WN6XQAQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX): protocol: expected piece hash; context canceled; EOF Oct 7 07:48:04.734238578 (330h16m50.196331061s) [noise=true , address=77.90.31.16:4756 , piece_id=4WCDI36U4YAC3C6KLZUHS44YRYIJ6BNDCO5VPP35JYKD4WN6XQAQ, seg_pos={0 0}, upload=290665]: Long tail cancellation due to optimal threshold Oct 7 07:48:04.734357075 (330h16m50.196449568s) [noise=true , address=45.91.116.6:4995 , piece_id=3IJGK3G2ZYWWRMDWOVXCNCEMAWP5YQXPNIBP5BM2LPV6GVL2H6NQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uaTGySSt8BWcdzfvw6aj2VrSvGG9xxMvTVvzn9fFA73AvxrxA): protocol: expected piece hash; context canceled; EOF Oct 7 07:48:04.734379557 (330h16m50.19647206s) [noise=true , address=77.90.31.14:4684 , piece_id=6VCBHVVFPETZFSFBXLBCTVPLXL3AZXSFRNOFWJTCYNW75C4F2WBA, seg_pos={0 0}, upload=290665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a7wpA6YchhigZCKC68YJzkBKwbqg1R1gBdfN8GdvrVregrK4m): protocol: expected piece hash; context canceled; EOF Oct 7 07:48:04.734393564 (330h16m50.196486047s) [noise=true , address=77.90.31.14:4684 , piece_id=6VCBHVVFPETZFSFBXLBCTVPLXL3AZXSFRNOFWJTCYNW75C4F2WBA, seg_pos={0 0}, upload=290665]: Long tail cancellation due to optimal threshold Oct 7 07:48:04.734403833 (330h16m50.196496316s) [noise=true , address=45.91.116.6:4995 , piece_id=3IJGK3G2ZYWWRMDWOVXCNCEMAWP5YQXPNIBP5BM2LPV6GVL2H6NQ, seg_pos={0 0}, upload=290665]: Long tail cancellation due to optimal threshold Oct 7 07:48:04.734509925 (330h16m50.196602418s) [noise=true , address=77.90.31.19:4863 , piece_id=2NVNF4CV7NQVKZQZ3QOUPSCNPETR4ZKHJRBVW6TJ4RAHMZBXH5CQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL): protocol: expected piece hash; context canceled; EOF Oct 7 07:48:04.734520886 (330h16m50.196613359s) [noise=true , address=77.90.31.19:4863 , piece_id=2NVNF4CV7NQVKZQZ3QOUPSCNPETR4ZKHJRBVW6TJ4RAHMZBXH5CQ, seg_pos={0 0}, upload=290665]: Long tail cancellation due to optimal threshold Oct 7 07:48:04.734627689 (330h16m50.196720172s) [seg_pos={0 0}, upload=290665]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 07:48:04.734735696 (330h16m50.196828179s) [upload=290665]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 07:48:04.734742689 (330h16m50.196835172s) [upload=290665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 07:49:01.566704025 (330h17m47.028796508s) [upload=290665]: Done uploading segments... Oct 7 07:49:01.566733461 (330h17m47.028825944s) [upload=290665]: Upload finished. err: <nil> Duration: 5.878811525s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.097627579s FAIL (2025-10-07 07:49:08 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== 2.684228524 (330h16m48.146321007s) [noise=true , address=145.79.227.22:4512 , piece_id=ZHYRXIBGXVLWMXK3JSTX5BM3KK3UYASIGFEA3HIKQEXWRPOXZKDA, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.684385934 (330h16m48.146478417s) [noise=true , address=45.91.116.14:4211 , piece_id=E6CAX7G3RVE22AJT7HOESRF633GR7YDDAHJLHOSYKXTMLUIFRGZA, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.684963844 (330h16m48.147056327s) [noise=true , address=77.90.31.14:4689 , piece_id=IVQS3GZNMI7JZFYVW3RHL4JSDVZ5UIHDNX5JEFV3346M3ZHQWNIQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.685221956 (330h16m48.147314439s) [noise=true , address=45.91.116.6:4995 , piece_id=3IJGK3G2ZYWWRMDWOVXCNCEMAWP5YQXPNIBP5BM2LPV6GVL2H6NQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.685452474 (330h16m48.147544957s) [noise=true , address=45.91.116.5:4302 , piece_id=DDYTKB75NQGJGPKPWC6AZ4RKDU6DBLA3AX2GNKEUUWJ227IDU5CQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.685900778 (330h16m48.147993261s) [noise=true , address=45.91.116.9:4272 , piece_id=EPDK76GOYFEJOFB4GJEB4WOKQ55L3PXKEJUQC66AUHE5BV3K3LBQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.685950814 (330h16m48.148043297s) [noise=true , address=45.91.116.12:4235 , piece_id=C3ZEP2EDRTNVHWMIH4OXWZVSACJRO3JO4FYMM5VTMGUWBFRSLRUA, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.686310157 (330h16m48.148402631s) [noise=true , address=77.90.31.10:4526 , piece_id=IZTN2WX2Q5P2ML4EZ2OGW23HQZ3UIXBQYWLUUTXJDFU73ZUELBMQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.686461365 (330h16m48.148553848s) [noise=true , address=77.90.31.16:4756 , piece_id=4WCDI36U4YAC3C6KLZUHS44YRYIJ6BNDCO5VPP35JYKD4WN6XQAQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.687006704 (330h16m48.149099187s) [noise=true , address=145.79.227.33:4900 , piece_id=5L3XYQELQU7KV7W466L5DCNXSP6U2LX7EIFOH6TUH67INZRGSWJQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.687022113 (330h16m48.149114596s) [noise=true , address=77.90.31.14:4684 , piece_id=6VCBHVVFPETZFSFBXLBCTVPLXL3AZXSFRNOFWJTCYNW75C4F2WBA, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.687498480 (330h16m48.149590963s) [noise=true , address=145.79.227.30:4777 , piece_id=UX4R7UC5EEX67KDL7SVKFPWD4VVB4WJPKAIBSRY5Y4IMAKRVKXYA, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.687559176 (330h16m48.149651699s) [noise=true , address=77.90.31.19:4863 , piece_id=2NVNF4CV7NQVKZQZ3QOUPSCNPETR4ZKHJRBVW6TJ4RAHMZBXH5CQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.688272183 (330h16m48.150364666s) [noise=true , address=77.90.31.21:4911 , piece_id=NOFTEDILTPXQ6LHAUDHP5GKZWI6J7VXGVJZNXYPZTUKQEZ5JO4OQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.690143407 (330h16m48.15223589s) [noise=true , address=145.79.227.29:4944 , piece_id=6IWBVDICAIROYEKIUBJZTSIZ6XI4KJHKY24F2L7BKHWGBUQA24LA, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.690606197 (330h16m48.152698671s) [noise=true , address=77.90.31.13:4643 , piece_id=UTPWYEFHE2B4BTPV7ANSJ7KUHYM4HF6NKRNPXX5NIAZQVRR3XDZQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.690968368 (330h16m48.153060851s) [noise=true , address=145.79.227.25:4639 , piece_id=SN5IMBBBZYXY4L24XQYZING6O2U7SBLFTPRWRU7EYAOGIGYG64IQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:04.184255931 (330h16m49.646348424s) [upload=290665]: Next returned nil segment Oct 7 07:48:04.184283844 (330h16m49.646376327s) [upload=290665]: Waiting for error group managing segments... Oct 7 07:48:04.558544184 (330h16m50.020636667s) [noise=true , address=145.79.227.22:4512 , piece_id=ZHYRXIBGXVLWMXK3JSTX5BM3KK3UYASIGFEA3HIKQEXWRPOXZKDA, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.559842245 (330h16m50.021934739s) [noise=true , address=145.79.227.30:4777 , piece_id=UX4R7UC5EEX67KDL7SVKFPWD4VVB4WJPKAIBSRY5Y4IMAKRVKXYA, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.567228070 (330h16m50.029320553s) [noise=true , address=145.79.227.23:4561 , piece_id=2UOAAEEW4AOLGFHNZTTFAHM3KJZTSDBO3MFCV2OMCN6TSJO46GZA, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.571144567 (330h16m50.03323706s) [noise=true , address=145.79.227.25:4639 , piece_id=SN5IMBBBZYXY4L24XQYZING6O2U7SBLFTPRWRU7EYAOGIGYG64IQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.571284323 (330h16m50.033376817s) [noise=true , address=145.79.227.29:4944 , piece_id=6IWBVDICAIROYEKIUBJZTSIZ6XI4KJHKY24F2L7BKHWGBUQA24LA, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.571824523 (330h16m50.033916996s) [noise=true , address=145.79.227.33:4900 , piece_id=5L3XYQELQU7KV7W466L5DCNXSP6U2LX7EIFOH6TUH67INZRGSWJQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.642083669 (330h16m50.104176162s) [noise=true , address=45.91.116.9:4272 , piece_id=EPDK76GOYFEJOFB4GJEB4WOKQ55L3PXKEJUQC66AUHE5BV3K3LBQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.652430721 (330h16m50.114523204s) [noise=true , address=45.91.116.12:4235 , piece_id=C3ZEP2EDRTNVHWMIH4OXWZVSACJRO3JO4FYMM5VTMGUWBFRSLRUA, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.660824113 (330h16m50.122916597s) [noise=true , address=77.90.31.21:4911 , piece_id=NOFTEDILTPXQ6LHAUDHP5GKZWI6J7VXGVJZNXYPZTUKQEZ5JO4OQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.680019976 (330h16m50.142112459s) [noise=true , address=45.91.116.14:4211 , piece_id=E6CAX7G3RVE22AJT7HOESRF633GR7YDDAHJLHOSYKXTMLUIFRGZA, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.705995618 (330h16m50.168088101s) [noise=true , address=77.90.31.14:4689 , piece_id=IVQS3GZNMI7JZFYVW3RHL4JSDVZ5UIHDNX5JEFV3346M3ZHQWNIQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.717852216 (330h16m50.179944699s) [noise=true , address=77.90.31.13:4643 , piece_id=UTPWYEFHE2B4BTPV7ANSJ7KUHYM4HF6NKRNPXX5NIAZQVRR3XDZQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.718777087 (330h16m50.18086957s) [noise=true , address=77.90.31.10:4526 , piece_id=IZTN2WX2Q5P2ML4EZ2OGW23HQZ3UIXBQYWLUUTXJDFU73ZUELBMQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.733559335 (330h16m50.195651819s) [noise=true , address=45.91.116.5:4302 , piece_id=DDYTKB75NQGJGPKPWC6AZ4RKDU6DBLA3AX2GNKEUUWJ227IDU5CQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.733600604 (330h16m50.195693087s) [seg_pos={0 0}, upload=290665]: Segment reached optimal threshold of 14 pieces. Oct 7 07:48:04.734223801 (330h16m50.196316294s) [noise=true , address=77.90.31.16:4756 , piece_id=4WCDI36U4YAC3C6KLZUHS44YRYIJ6BNDCO5VPP35JYKD4WN6XQAQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX): protocol: expected piece hash; context canceled; EOF Oct 7 07:48:04.734238578 (330h16m50.196331061s) [noise=true , address=77.90.31.16:4756 , piece_id=4WCDI36U4YAC3C6KLZUHS44YRYIJ6BNDCO5VPP35JYKD4WN6XQAQ, seg_pos={0 0}, upload=290665]: Long tail cancellation due to optimal threshold Oct 7 07:48:04.734357075 (330h16m50.196449568s) [noise=true , address=45.91.116.6:4995 , piece_id=3IJGK3G2ZYWWRMDWOVXCNCEMAWP5YQXPNIBP5BM2LPV6GVL2H6NQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uaTGySSt8BWcdzfvw6aj2VrSvGG9xxMvTVvzn9fFA73AvxrxA): protocol: expected piece hash; context canceled; EOF Oct 7 07:48:04.734379557 (330h16m50.19647206s) [noise=true , address=77.90.31.14:4684 , piece_id=6VCBHVVFPETZFSFBXLBCTVPLXL3AZXSFRNOFWJTCYNW75C4F2WBA, seg_pos={0 0}, upload=290665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a7wpA6YchhigZCKC68YJzkBKwbqg1R1gBdfN8GdvrVregrK4m): protocol: expected piece hash; context canceled; EOF Oct 7 07:48:04.734393564 (330h16m50.196486047s) [noise=true , address=77.90.31.14:4684 , piece_id=6VCBHVVFPETZFSFBXLBCTVPLXL3AZXSFRNOFWJTCYNW75C4F2WBA, seg_pos={0 0}, upload=290665]: Long tail cancellation due to optimal threshold Oct 7 07:48:04.734403833 (330h16m50.196496316s) [noise=true , address=45.91.116.6:4995 , piece_id=3IJGK3G2ZYWWRMDWOVXCNCEMAWP5YQXPNIBP5BM2LPV6GVL2H6NQ, seg_pos={0 0}, upload=290665]: Long tail cancellation due to optimal threshold Oct 7 07:48:04.734509925 (330h16m50.196602418s) [noise=true , address=77.90.31.19:4863 , piece_id=2NVNF4CV7NQVKZQZ3QOUPSCNPETR4ZKHJRBVW6TJ4RAHMZBXH5CQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL): protocol: expected piece hash; context canceled; EOF Oct 7 07:48:04.734520886 (330h16m50.196613359s) [noise=true , address=77.90.31.19:4863 , piece_id=2NVNF4CV7NQVKZQZ3QOUPSCNPETR4ZKHJRBVW6TJ4RAHMZBXH5CQ, seg_pos={0 0}, upload=290665]: Long tail cancellation due to optimal threshold Oct 7 07:48:04.734627689 (330h16m50.196720172s) [seg_pos={0 0}, upload=290665]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 07:48:04.734735696 (330h16m50.196828179s) [upload=290665]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 07:48:04.734742689 (330h16m50.196835172s) [upload=290665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 07:49:01.566704025 (330h17m47.028796508s) [upload=290665]: Done uploading segments... Oct 7 07:49:01.566733461 (330h17m47.028825944s) [upload=290665]: Upload finished. err: <nil> Duration: 6.097655123s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.210742682s FAIL (2025-10-07 07:49:07 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.210766637s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.002748696s FAIL (2025-10-07 07:49:01 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== 2.684228524 (330h16m48.146321007s) [noise=true , address=145.79.227.22:4512 , piece_id=ZHYRXIBGXVLWMXK3JSTX5BM3KK3UYASIGFEA3HIKQEXWRPOXZKDA, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.684385934 (330h16m48.146478417s) [noise=true , address=45.91.116.14:4211 , piece_id=E6CAX7G3RVE22AJT7HOESRF633GR7YDDAHJLHOSYKXTMLUIFRGZA, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.684963844 (330h16m48.147056327s) [noise=true , address=77.90.31.14:4689 , piece_id=IVQS3GZNMI7JZFYVW3RHL4JSDVZ5UIHDNX5JEFV3346M3ZHQWNIQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.685221956 (330h16m48.147314439s) [noise=true , address=45.91.116.6:4995 , piece_id=3IJGK3G2ZYWWRMDWOVXCNCEMAWP5YQXPNIBP5BM2LPV6GVL2H6NQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.685452474 (330h16m48.147544957s) [noise=true , address=45.91.116.5:4302 , piece_id=DDYTKB75NQGJGPKPWC6AZ4RKDU6DBLA3AX2GNKEUUWJ227IDU5CQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.685900778 (330h16m48.147993261s) [noise=true , address=45.91.116.9:4272 , piece_id=EPDK76GOYFEJOFB4GJEB4WOKQ55L3PXKEJUQC66AUHE5BV3K3LBQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.685950814 (330h16m48.148043297s) [noise=true , address=45.91.116.12:4235 , piece_id=C3ZEP2EDRTNVHWMIH4OXWZVSACJRO3JO4FYMM5VTMGUWBFRSLRUA, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.686310157 (330h16m48.148402631s) [noise=true , address=77.90.31.10:4526 , piece_id=IZTN2WX2Q5P2ML4EZ2OGW23HQZ3UIXBQYWLUUTXJDFU73ZUELBMQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.686461365 (330h16m48.148553848s) [noise=true , address=77.90.31.16:4756 , piece_id=4WCDI36U4YAC3C6KLZUHS44YRYIJ6BNDCO5VPP35JYKD4WN6XQAQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.687006704 (330h16m48.149099187s) [noise=true , address=145.79.227.33:4900 , piece_id=5L3XYQELQU7KV7W466L5DCNXSP6U2LX7EIFOH6TUH67INZRGSWJQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.687022113 (330h16m48.149114596s) [noise=true , address=77.90.31.14:4684 , piece_id=6VCBHVVFPETZFSFBXLBCTVPLXL3AZXSFRNOFWJTCYNW75C4F2WBA, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.687498480 (330h16m48.149590963s) [noise=true , address=145.79.227.30:4777 , piece_id=UX4R7UC5EEX67KDL7SVKFPWD4VVB4WJPKAIBSRY5Y4IMAKRVKXYA, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.687559176 (330h16m48.149651699s) [noise=true , address=77.90.31.19:4863 , piece_id=2NVNF4CV7NQVKZQZ3QOUPSCNPETR4ZKHJRBVW6TJ4RAHMZBXH5CQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.688272183 (330h16m48.150364666s) [noise=true , address=77.90.31.21:4911 , piece_id=NOFTEDILTPXQ6LHAUDHP5GKZWI6J7VXGVJZNXYPZTUKQEZ5JO4OQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.690143407 (330h16m48.15223589s) [noise=true , address=145.79.227.29:4944 , piece_id=6IWBVDICAIROYEKIUBJZTSIZ6XI4KJHKY24F2L7BKHWGBUQA24LA, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.690606197 (330h16m48.152698671s) [noise=true , address=77.90.31.13:4643 , piece_id=UTPWYEFHE2B4BTPV7ANSJ7KUHYM4HF6NKRNPXX5NIAZQVRR3XDZQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:02.690968368 (330h16m48.153060851s) [noise=true , address=145.79.227.25:4639 , piece_id=SN5IMBBBZYXY4L24XQYZING6O2U7SBLFTPRWRU7EYAOGIGYG64IQ, seg_pos={0 0}, upload=290665]: Uploading piece... Oct 7 07:48:04.184255931 (330h16m49.646348424s) [upload=290665]: Next returned nil segment Oct 7 07:48:04.184283844 (330h16m49.646376327s) [upload=290665]: Waiting for error group managing segments... Oct 7 07:48:04.558544184 (330h16m50.020636667s) [noise=true , address=145.79.227.22:4512 , piece_id=ZHYRXIBGXVLWMXK3JSTX5BM3KK3UYASIGFEA3HIKQEXWRPOXZKDA, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.559842245 (330h16m50.021934739s) [noise=true , address=145.79.227.30:4777 , piece_id=UX4R7UC5EEX67KDL7SVKFPWD4VVB4WJPKAIBSRY5Y4IMAKRVKXYA, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.567228070 (330h16m50.029320553s) [noise=true , address=145.79.227.23:4561 , piece_id=2UOAAEEW4AOLGFHNZTTFAHM3KJZTSDBO3MFCV2OMCN6TSJO46GZA, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.571144567 (330h16m50.03323706s) [noise=true , address=145.79.227.25:4639 , piece_id=SN5IMBBBZYXY4L24XQYZING6O2U7SBLFTPRWRU7EYAOGIGYG64IQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.571284323 (330h16m50.033376817s) [noise=true , address=145.79.227.29:4944 , piece_id=6IWBVDICAIROYEKIUBJZTSIZ6XI4KJHKY24F2L7BKHWGBUQA24LA, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.571824523 (330h16m50.033916996s) [noise=true , address=145.79.227.33:4900 , piece_id=5L3XYQELQU7KV7W466L5DCNXSP6U2LX7EIFOH6TUH67INZRGSWJQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.642083669 (330h16m50.104176162s) [noise=true , address=45.91.116.9:4272 , piece_id=EPDK76GOYFEJOFB4GJEB4WOKQ55L3PXKEJUQC66AUHE5BV3K3LBQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.652430721 (330h16m50.114523204s) [noise=true , address=45.91.116.12:4235 , piece_id=C3ZEP2EDRTNVHWMIH4OXWZVSACJRO3JO4FYMM5VTMGUWBFRSLRUA, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.660824113 (330h16m50.122916597s) [noise=true , address=77.90.31.21:4911 , piece_id=NOFTEDILTPXQ6LHAUDHP5GKZWI6J7VXGVJZNXYPZTUKQEZ5JO4OQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.680019976 (330h16m50.142112459s) [noise=true , address=45.91.116.14:4211 , piece_id=E6CAX7G3RVE22AJT7HOESRF633GR7YDDAHJLHOSYKXTMLUIFRGZA, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.705995618 (330h16m50.168088101s) [noise=true , address=77.90.31.14:4689 , piece_id=IVQS3GZNMI7JZFYVW3RHL4JSDVZ5UIHDNX5JEFV3346M3ZHQWNIQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.717852216 (330h16m50.179944699s) [noise=true , address=77.90.31.13:4643 , piece_id=UTPWYEFHE2B4BTPV7ANSJ7KUHYM4HF6NKRNPXX5NIAZQVRR3XDZQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.718777087 (330h16m50.18086957s) [noise=true , address=77.90.31.10:4526 , piece_id=IZTN2WX2Q5P2ML4EZ2OGW23HQZ3UIXBQYWLUUTXJDFU73ZUELBMQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.733559335 (330h16m50.195651819s) [noise=true , address=45.91.116.5:4302 , piece_id=DDYTKB75NQGJGPKPWC6AZ4RKDU6DBLA3AX2GNKEUUWJ227IDU5CQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: <nil> Oct 7 07:48:04.733600604 (330h16m50.195693087s) [seg_pos={0 0}, upload=290665]: Segment reached optimal threshold of 14 pieces. Oct 7 07:48:04.734223801 (330h16m50.196316294s) [noise=true , address=77.90.31.16:4756 , piece_id=4WCDI36U4YAC3C6KLZUHS44YRYIJ6BNDCO5VPP35JYKD4WN6XQAQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX): protocol: expected piece hash; context canceled; EOF Oct 7 07:48:04.734238578 (330h16m50.196331061s) [noise=true , address=77.90.31.16:4756 , piece_id=4WCDI36U4YAC3C6KLZUHS44YRYIJ6BNDCO5VPP35JYKD4WN6XQAQ, seg_pos={0 0}, upload=290665]: Long tail cancellation due to optimal threshold Oct 7 07:48:04.734357075 (330h16m50.196449568s) [noise=true , address=45.91.116.6:4995 , piece_id=3IJGK3G2ZYWWRMDWOVXCNCEMAWP5YQXPNIBP5BM2LPV6GVL2H6NQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uaTGySSt8BWcdzfvw6aj2VrSvGG9xxMvTVvzn9fFA73AvxrxA): protocol: expected piece hash; context canceled; EOF Oct 7 07:48:04.734379557 (330h16m50.19647206s) [noise=true , address=77.90.31.14:4684 , piece_id=6VCBHVVFPETZFSFBXLBCTVPLXL3AZXSFRNOFWJTCYNW75C4F2WBA, seg_pos={0 0}, upload=290665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a7wpA6YchhigZCKC68YJzkBKwbqg1R1gBdfN8GdvrVregrK4m): protocol: expected piece hash; context canceled; EOF Oct 7 07:48:04.734393564 (330h16m50.196486047s) [noise=true , address=77.90.31.14:4684 , piece_id=6VCBHVVFPETZFSFBXLBCTVPLXL3AZXSFRNOFWJTCYNW75C4F2WBA, seg_pos={0 0}, upload=290665]: Long tail cancellation due to optimal threshold Oct 7 07:48:04.734403833 (330h16m50.196496316s) [noise=true , address=45.91.116.6:4995 , piece_id=3IJGK3G2ZYWWRMDWOVXCNCEMAWP5YQXPNIBP5BM2LPV6GVL2H6NQ, seg_pos={0 0}, upload=290665]: Long tail cancellation due to optimal threshold Oct 7 07:48:04.734509925 (330h16m50.196602418s) [noise=true , address=77.90.31.19:4863 , piece_id=2NVNF4CV7NQVKZQZ3QOUPSCNPETR4ZKHJRBVW6TJ4RAHMZBXH5CQ, seg_pos={0 0}, upload=290665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL): protocol: expected piece hash; context canceled; EOF Oct 7 07:48:04.734520886 (330h16m50.196613359s) [noise=true , address=77.90.31.19:4863 , piece_id=2NVNF4CV7NQVKZQZ3QOUPSCNPETR4ZKHJRBVW6TJ4RAHMZBXH5CQ, seg_pos={0 0}, upload=290665]: Long tail cancellation due to optimal threshold Oct 7 07:48:04.734627689 (330h16m50.196720172s) [seg_pos={0 0}, upload=290665]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 07:48:04.734735696 (330h16m50.196828179s) [upload=290665]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 07:48:04.734742689 (330h16m50.196835172s) [upload=290665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 07:49:01.566704025 (330h17m47.028796508s) [upload=290665]: Done uploading segments... Oct 7 07:49:01.566733461 (330h17m47.028825944s) [upload=290665]: Upload finished. err: <nil> Duration: 1m0.002782752s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 58.415980371s FAIL (2025-10-07 07:49:00 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 58.416017802s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.167189261s FAIL (2025-10-07 07:49:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.167219619s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001538682s FAIL (2025-10-07 07:48:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001568478s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.808491726s FAIL (2025-10-07 07:37:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 5.808525271s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 5.94081768s FAIL (2025-10-07 07:37:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 5.940847537s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 5.872466508s FAIL (2025-10-07 07:36:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 5.872503248s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [XXXX........................................................] duration: 6.046508354s FAIL (2025-10-07 07:36:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 6.046541869s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.XXXX.......................................................] duration: 5.998329262s FAIL (2025-10-07 07:35:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 5.998365022s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.XXXX......................................................] duration: 5.972596545s FAIL (2025-10-07 07:33:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 5.972702978s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [XX.XXXX.....................................................] duration: 5.790185028s FAIL (2025-10-07 07:32:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 5.790214445s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.22555886s FAIL (2025-10-07 07:32:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.225598374s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.021925703s FAIL (2025-10-07 07:32:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.021962144s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [XXX.XXXX....................................................] duration: 5.695850166s FAIL (2025-10-07 07:31:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 5.695881645s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.X.........................................................] duration: 6.097895956s FAIL (2025-10-07 07:31:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.097944841s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.XXX.XXXX...................................................] duration: 6.129771631s FAIL (2025-10-07 07:31:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 6.129798704s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [XX.X........................................................] duration: 5.98568246s FAIL (2025-10-07 07:30:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 5.985744748s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.XXX.XXXX..................................................] duration: 5.918344596s FAIL (2025-10-07 07:30:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 5.918374544s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [XXX.X.......................................................] duration: 6.091011988s FAIL (2025-10-07 07:30:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.091042726s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X.XXX.XXXX.................................................] duration: 5.872221479s FAIL (2025-10-07 07:29:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 5.872264621s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.XXX.X......................................................] duration: 5.934245796s FAIL (2025-10-07 07:29:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 5.934282276s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X.XXX.XXXX................................................] duration: 5.739381217s FAIL (2025-10-07 07:28:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 5.739416784s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.XXX.X.....................................................] duration: 6.211229052s FAIL (2025-10-07 07:28:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.211311217s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X.XXX.XXXX...............................................] duration: 5.673794308s FAIL (2025-10-07 07:27:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 5.673830707s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [XX.XXX.X....................................................] duration: 5.960944624s FAIL (2025-10-07 07:27:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 5.960975763s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X.XXX.XXXX..............................................] duration: 5.787400716s FAIL (2025-10-07 07:27:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 5.787430773s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [XXX.XXX.X...................................................] duration: 5.866662834s FAIL (2025-10-07 07:26:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 5.866692249s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [XXXX.XXX.X..................................................] duration: 6.230204857s FAIL (2025-10-07 07:26:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.230235635s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [X....X.XXX.XXXX.............................................] duration: 5.739708789s FAIL (2025-10-07 07:25:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 5.739743265s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [XXXXX.XXX.X.................................................] duration: 6.22968703s FAIL (2025-10-07 07:25:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.229717558s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X....X.XXX.XXXX............................................] duration: 6.099222109s FAIL (2025-10-07 07:24:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 6.099251415s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [XXXXXX.XXX.X................................................] duration: 6.358181701s FAIL (2025-10-07 07:24:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.358214703s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X....X.XXX.XXXX...........................................] duration: 5.975798794s FAIL (2025-10-07 07:24:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 5.975862596s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X....X.XXX.XXXX..........................................] duration: 6.454474883s FAIL (2025-10-07 07:23:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 6.454502475s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [XXXXXXX.XXX.X...............................................] duration: 6.175776292s FAIL (2025-10-07 07:23:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.17580693s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XXXXXXXX.XXX.X..............................................] duration: 6.976353413s FAIL (2025-10-07 07:23:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.976386115s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.XXXXXXXX.XXX.X.............................................] duration: 6.20508308s FAIL (2025-10-07 07:22:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.205115243s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [....X....X.XXX.XXXX.........................................] duration: 6.012514791s FAIL (2025-10-07 07:22:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 6.012550679s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..XXXXXXXX.XXX.X............................................] duration: 7.068402429s FAIL (2025-10-07 07:22:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 7.068432448s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....X....X.XXX.XXXX........................................] duration: 6.732692251s FAIL (2025-10-07 07:21:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 6.732721047s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...XXXXXXXX.XXX.X...........................................] duration: 6.609659163s FAIL (2025-10-07 07:21:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.609682588s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [......X....X.XXX.XXXX.......................................] duration: 6.05568242s FAIL (2025-10-07 07:21:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 6.055706084s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [.......X....X.XXX.XXXX......................................] duration: 6.198457305s FAIL (2025-10-07 07:20:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 6.198486409s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....XXXXXXXX.XXX.X..........................................] duration: 6.197282725s FAIL (2025-10-07 07:20:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.19731165s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [........X....X.XXX.XXXX.....................................] duration: 5.890984845s FAIL (2025-10-07 07:20:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 5.891014392s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....XXXXXXXX.XXX.X.........................................] duration: 6.165519261s FAIL (2025-10-07 07:19:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ) piecestore close: write tcp 188.166.225.170:43094->77.90.31.17:4770: use of closed network connection; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA) context deadline exceeded ======LOGS====== ad=289831]: Starting upload Oct 7 06:51:50.235180289 (329h20m35.697272762s) [upload=289831]: Done starting upload Oct 7 06:51:50.235360752 (329h20m35.697453235s) [upload=289831]: Uploading segments... Oct 7 06:51:50.235478376 (329h20m35.697570859s) [upload=289831]: Got next segment. Inline: false Oct 7 06:51:50.235526288 (329h20m35.697618761s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 7 06:51:50.235530836 (329h20m35.697623319s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 7 06:51:50.489386760 (329h20m35.951479243s) [seg_pos={0 0}, upload=289831]: Begin upload segment... Oct 7 06:51:50.489446053 (329h20m35.951538537s) [seg_pos={0 0}, upload=289831]: Done begin upload segment. Oct 7 06:51:50.489472454 (329h20m35.951564927s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489483775 (329h20m35.951576248s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.489946706 (329h20m35.952039179s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490539956 (329h20m35.952632429s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.490654204 (329h20m35.952746687s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491086637 (329h20m35.95317911s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491158584 (329h20m35.953251077s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491426685 (329h20m35.953519158s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491618901 (329h20m35.953711384s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491733990 (329h20m35.953826473s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.491983616 (329h20m35.954076089s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492051896 (329h20m35.954144379s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492318544 (329h20m35.954411027s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492390821 (329h20m35.954483294s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.492812815 (329h20m35.954905298s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:50.493177759 (329h20m35.955270242s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Uploading piece... Oct 7 06:51:52.173664983 (329h20m37.635757466s) [upload=289831]: Next returned nil segment Oct 7 06:51:52.173686854 (329h20m37.635779337s) [upload=289831]: Waiting for error group managing segments... Oct 7 06:51:52.559486130 (329h20m38.021578613s) [noise=true , address=45.91.116.7:4011 , piece_id=QU6O55VDVCJQPU75OQ7ZWATTMRXV6PN4UWLHK5ALZPIAMCV6KKNA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.563137593 (329h20m38.025230076s) [noise=true , address=77.90.31.21:4877 , piece_id=U3JLFNUJ43KZY6U7KEY6XDG6AIIWTURWBJ3TFT7UIRV72JBEFX3A, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.568473153 (329h20m38.030565636s) [noise=true , address=45.91.116.13:4155 , piece_id=HHE5DWD357RSDPTODHWC6BFR3TT66OW3GPLODXN35CT7W67I6TGA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.571556784 (329h20m38.033649268s) [noise=true , address=77.90.31.17:4770 , piece_id=E3LUCR27SIFLAIPKRLC7KY553VRFNYBVWAFLYCLMASO3MYKZ6MPQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.610893538 (329h20m38.072986011s) [noise=true , address=45.91.116.12:4225 , piece_id=AGJWX4QX2VA2TZP36TYRH3ZA2Y6T4XSH5SIHM4IB4MMZQFFTHJMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.637723106 (329h20m38.099815599s) [noise=true , address=77.90.31.13:4627 , piece_id=XPICPS43T7OSSL4YUCSWUR255K2NRW2JS3VUR5LZBJ5TZ6NF3LMA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.638664629 (329h20m38.100757112s) [noise=true , address=77.90.31.16:4734 , piece_id=N2R6ZMYWWYKKBYVCANPTOG2P2KGLIYCMDGKFDJZFA33STMSFT6GQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.671457492 (329h20m38.133549975s) [noise=true , address=45.91.116.13:4154 , piece_id=4CTWENWVA33KNZZVX6D2EIHCEXW7NDFEK6WJURYVROTUB2PY4UHQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.683291536 (329h20m38.145384019s) [noise=true , address=45.91.116.14:4189 , piece_id=BDOAKV2BCAFX6YJGKO5WVSQBEMIPW2PFI7J4DSJZU7JMXRGR3NOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.693680980 (329h20m38.155773463s) [noise=true , address=77.90.31.10:4519 , piece_id=T2HJJKNIRP3XRG4OF5P5RERIFYL4KQXL25LT76WV7XZQ4CO72BOA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.730622553 (329h20m38.192715036s) [noise=true , address=45.91.116.5:4299 , piece_id=IMTJ5NPLVY6FDHRPIILEBNYEU4CVCXBW2VEEXICIZS5FAE6XTEYA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.734749601 (329h20m38.196842084s) [noise=true , address=45.91.116.5:4297 , piece_id=2WP5Z4RSLVO553UEWNYVR5EUMNI35LNQUFSZUETILZXEMMKDCA3Q, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784458485 (329h20m38.246550968s) [noise=true , address=45.91.116.11:4117 , piece_id=BJ3VSI3BGZ2DVCMM4G4HSDJRT3LKPIMHUOT62WNI3HLMQIB3R6MA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: <nil> Oct 7 06:51:52.784487500 (329h20m38.246579984s) [seg_pos={0 0}, upload=289831]: Segment reached optimal threshold of 13 pieces. Oct 7 06:51:52.785353379 (329h20m38.247445872s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: rpc: dial tcp 145.79.227.2:4115: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785369680 (329h20m38.247462163s) [noise=true , address=145.79.227.2:4115 , piece_id=QNLWYEEK6KKEZ5F7HHMHJJMS6CFOVEQLVOV3WQF63WFBDCLLA7UA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785579379 (329h20m38.247671862s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: read tcp 188.166.225.170:47296->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785591773 (329h20m38.247684257s) [noise=true , address=145.79.227.16:4910 , piece_id=L4SVDIJX27ZY4IL6JZ2PHWMLBMJA7YGLGC6J5BYSAT64WF4UDYTA, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785726520 (329h20m38.247819013s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: read tcp 188.166.225.170:36690->145.79.227.19:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:51:52.785735808 (329h20m38.247828291s) [noise=true , address=145.79.227.19:4078 , piece_id=TYNHZIMXTHXGBIWEMUPCRD2C2EXHOPBLCV2CUDV4OHSIDFTOQXZQ, seg_pos={0 0}, upload=289831]: Long tail cancellation due to optimal threshold Oct 7 06:51:52.785754523 (329h20m38.247847006s) [seg_pos={0 0}, upload=289831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 7 06:51:52.786067308 (329h20m38.248159781s) [upload=289831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:51:52.786077008 (329h20m38.248169491s) [upload=289831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:51:53.221867805 (329h20m38.683960288s) [upload=289831]: Done uploading segments... Oct 7 06:51:53.221893353 (329h20m38.683985837s) [upload=289831]: Upload finished. err: <nil> Duration: 6.165546812s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [.........X....X.XXX.XXXX....................................] duration: 5.921170391s FAIL (2025-10-07 07:19:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 0m13.663786069s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.201972678 (329h20m13.664065151s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202214568 (329h20m13.664307041s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202477639 (329h20m13.664570112s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202719019 (329h20m13.664811492s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.202954858 (329h20m13.665047341s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203195988 (329h20m13.665288471s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203444370 (329h20m13.665536843s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.203709945 (329h20m13.665802429s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204029956 (329h20m13.666122429s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204293427 (329h20m13.6663859s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204564904 (329h20m13.666657387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.204897538 (329h20m13.666990011s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205219200 (329h20m13.667311683s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:28.205527277 (329h20m13.66761975s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Uploading piece... Oct 7 06:51:29.736522626 (329h20m15.198615099s) [upload=289826]: Next returned nil segment Oct 7 06:51:29.736541020 (329h20m15.198633503s) [upload=289826]: Waiting for error group managing segments... Oct 7 06:51:30.125616866 (329h20m15.587709349s) [noise=true , address=77.90.31.13:4645 , piece_id=SXWFSXCXGCJ6JQQ3TLUSXISLZOZPZ5F7G27YMMWJR2HS4LIACQRQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.137671500 (329h20m15.599763983s) [noise=true , address=45.91.116.5:4314 , piece_id=ELMJ3WLDCD3O3DU34HQBXHZK363PRFIEUOHDRR3MF5R23HP2KIVQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.171741454 (329h20m15.633833927s) [noise=true , address=45.91.116.4:4942 , piece_id=T63PJRNBHMZBHRCNVAXIP2MVYADHNY7CWYMY5R5PWTOBOORRTZEA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.193322979 (329h20m15.655415462s) [noise=true , address=45.91.116.4:4971 , piece_id=3DLHYCS4KUFQHCCTIOFTMKABKV7H5LB5K3UFSBJ7I4AAQBS5ONSQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.209701726 (329h20m15.67179421s) [noise=true , address=45.91.116.7:4020 , piece_id=PZUL75P6KSD4L7JAVLFK2LDCX6GHDJZ763XIN256QHG5T7UA7S6Q, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.218096993 (329h20m15.680189476s) [noise=true , address=77.90.31.10:4542 , piece_id=UTXGCBORDPRLX3KMCA6MQ5JQZOAYHBZ67MVZM7MRKC3745KRJPTQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.249328143 (329h20m15.711420636s) [noise=true , address=45.91.116.6:4993 , piece_id=SWWNGUZTY2VVLZ4SRX4TG62HN3ASKZIAO4ITM6ZIZBKAG24YBVZQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.300300332 (329h20m15.762392815s) [noise=true , address=77.90.31.15:4732 , piece_id=N3K6T6SBE4QULSC3NNQAP2IGPKMY3F3TPDF2WHB5O6T2JDRAQKZA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:30.612615722 (329h20m16.074708205s) [noise=true , address=77.90.31.18:4823 , piece_id=L6RW4RMRJJNIDEGVH4EXCUHTGWM4EBZG4K4DQAJCQPFKADWZGGIA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:51:31.000131546 (329h20m16.462224039s) [noise=true , address=77.90.31.14:4681 , piece_id=PI3OENRLUYB3P2CXJ3UDBV3DSPFWKIUIMEJS5VKWNJQWLHP2M37A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:21.297772550 (329h21m6.759865033s) [noise=true , address=145.79.227.23:4565 , piece_id=GDGUAZTT3AAZTIW67VTX6GWBS7ANAQ3RQ6KNAXYHY4RBB7HYACNQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.477208506 (329h21m8.939300989s) [noise=true , address=145.79.227.28:4746 , piece_id=TKE37ED7JSCLOSTTXBQLT4W4GXYIRYOTUBUE2LHVZKCHIU7WBDAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.656318130 (329h21m9.118410613s) [noise=true , address=145.79.227.30:4788 , piece_id=CJTC7BNPJE5EWNFHGTDWTIYW46LO5FYG5NZFESHGL2X4U4GYIUPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817377904 (329h21m9.279470387s) [noise=true , address=145.79.227.33:4897 , piece_id=MJMUATPYYOLIGIFQN6WATJGLROHKKN6WJAKDEO5QDDZ5ZRX3PWPA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: <nil> Oct 7 06:52:23.817425264 (329h21m9.279517777s) [seg_pos={0 0}, upload=289826]: Segment reached optimal threshold of 14 pieces. Oct 7 06:52:23.817666694 (329h21m9.279759187s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD): protocol: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59964->145.79.227.32:4841: use of closed network connection Oct 7 06:52:23.817688254 (329h21m9.279780737s) [noise=true , address=145.79.227.32:4841 , piece_id=E6XSNV5BTKLRLRF3RGN2TX2Z33644MM46WDUFKSCAXG25CTPNS3A, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817768908 (329h21m9.279861391s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj): protocol: read tcp 188.166.225.170:38192->145.79.227.44:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.817780440 (329h21m9.279872923s) [noise=true , address=145.79.227.44:4200 , piece_id=C5DTAOCSU37PPYECFP7DGZMJC7AJRB7XHDNGA6QQ5IZSWOA44WAQ, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817898866 (329h21m9.279991339s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWzLpDf6jtpMcAFyF4WWUgcWV6cJMKNWRo6wZYvtZ1rDoQ1Hh): protocol: context canceled; context canceled Oct 7 06:52:23.817907172 (329h21m9.279999645s) [noise=true , address=145.79.227.31:4820 , piece_id=UHW6SB4UM4ZNTVOMNOTWEAQDZ5WJWL3NTWNGEB4KOHQNSOCTTJVA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.817998645 (329h21m9.280091118s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: read tcp 188.166.225.170:49366->145.79.227.49:4356: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 7 06:52:23.818007192 (329h21m9.280099675s) [noise=true , address=145.79.227.49:4356 , piece_id=WOSFNISNO6JBO4NFK43I666MZ3NVLFG3EWZPEWU7UUD57UXAVISA, seg_pos={0 0}, upload=289826]: Long tail cancellation due to optimal threshold Oct 7 06:52:23.818027320 (329h21m9.280119803s) [seg_pos={0 0}, upload=289826]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 7 06:52:23.818099567 (329h21m9.280192051s) [upload=289826]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 7 06:52:23.818104236 (329h21m9.280196709s) [upload=289826]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 7 06:52:24.148601679 (329h21m9.610694162s) [upload=289826]: Done uploading segments... Oct 7 06:52:24.148623171 (329h21m9.610715644s) [upload=289826]: Upload finished. err: <nil> Duration: 5.921267827s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)