Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X..................................X...............] duration: 15.168104974s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 08:51:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 15.168130403s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X..................................X..............] duration: 25.439842484s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 08:51:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 25.43986075s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...X.X..X...X..X...X...X...X.....XXX......X....X..XXX..X....] duration: 5.924458755s trace: https://jaeger.dev.storj.io/trace/5d1cfb69a266dd0b FAIL (2025-12-29 08:50:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 29 08:46:47.371802386 (569h17m47.242885532s) [upload=499115]: Got next segment. Inline: false Dec 29 08:46:47.371866869 (569h17m47.242949935s) [upload=499115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 08:46:47.371893961 (569h17m47.242977036s) [upload=499115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 08:46:47.619452822 (569h17m47.490535868s) [seg_pos={0 0}, upload=499115]: Begin upload segment... Dec 29 08:46:47.619498429 (569h17m47.490581464s) [seg_pos={0 0}, upload=499115]: Done begin upload segment. Dec 29 08:46:47.619545328 (569h17m47.490628363s) [noise=true , address=45.91.116.14:4190 , piece_id=455VBRBSG6BKGVG2LXOEOSHEXREZOZD6YIOSJ2NETLQRPOSWIAPA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.619604240 (569h17m47.490687286s) [noise=true , address=77.90.31.19:4844 , piece_id=ZAECYYMVBYNVRJMKDUZOQQX75NSRP5MEC4FN65HA7OFVUTTCVL4Q, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.620583435 (569h17m47.49166647s) [noise=true , address=77.90.31.10:4517 , piece_id=CUPF4TEOOKGJUXTWWJC56O7RUQBQRGGHIVIFFUI7ZX3QLLDL4ERA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.620988637 (569h17m47.492071672s) [noise=true , address=145.79.227.52:4445 , piece_id=IEXXJ7RP5524JUHQBESOOM2CXUDCINICHIIPIRTZW2BVDOANKECQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.621302364 (569h17m47.492385399s) [noise=true , address=45.91.116.12:4228 , piece_id=HMCUUC5ATKK6QLR47TQVZ74HGALA7UAQZ5Q42RTVLIOBIZ6Q7YUA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.621613196 (569h17m47.492696241s) [noise=true , address=77.90.31.17:4773 , piece_id=LPQLAK64IRWJURQBFFNKWBEOHZND4TGRFQZSWKFEPKG5755XMGIA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622000353 (569h17m47.493083398s) [noise=true , address=77.90.31.15:4697 , piece_id=GSYH4OXG4YUKGT7GMSMLLBPT6EACKAZGXCLQ2KE5XJR4KYEGZAWA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622046350 (569h17m47.493129386s) [noise=true , address=145.79.227.38:4978 , piece_id=R5TZIOYZIB3TJVAWV2JN6RLQ6IAN2ZSDU2YZVW4OIIDSTR7IP2JQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622289293 (569h17m47.493372328s) [noise=true , address=145.79.227.48:4303 , piece_id=SBUUIIPD4YEEYHUHHY46UJLVD5YBMTUSAO54QCTSFPNKD5YNV6IQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622478693 (569h17m47.493561728s) [noise=true , address=145.79.227.5:4221 , piece_id=THB3MJWYUU6EB4O6DUJ33V644WP247CF5N4UBEKV2OTWDELE5NRA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622595486 (569h17m47.493678531s) [noise=true , address=145.79.227.22:4509 , piece_id=FGKOGHI3SFNGEEBUAC2MU2CHI7LR2A6PUQEAATZ64IYD2CEVOPQQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622948328 (569h17m47.494031363s) [noise=true , address=145.79.227.45:4230 , piece_id=FGWGYTFTQOIGWL46UIPMRXVY27HAZBHFO6TQAXFWMK47SLWTH5GQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622999846 (569h17m47.494082881s) [noise=true , address=145.79.227.41:4086 , piece_id=HSZKIFSMXZEYHJWPORVJI7V4VLL5Z4CQYGMFHT55GQCTPNXDG6IA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.623421570 (569h17m47.494504615s) [noise=true , address=45.91.116.7:4010 , piece_id=BIRGU2P5CFK2L7OWE7QXFHROVBTFQKB6QM3SW5IKF5H3RJH3UDSQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.623927052 (569h17m47.495010087s) [noise=true , address=145.79.227.36:4517 , piece_id=UQUYFUJN27A2XFPNOTEKNUJCP4JCKDFQ7HR4U2HJBCPC4SDVEO2Q, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.624227705 (569h17m47.49531074s) [noise=true , address=145.79.227.9:4367 , piece_id=7YH3VPUEPA6CQHFJ6MZAGGSJ3KYVH623ZUPB3KLWZNMT3CIHJF6Q, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:49.328824315 (569h17m49.19990736s) [upload=499115]: Next returned nil segment Dec 29 08:46:49.328859142 (569h17m49.199942197s) [upload=499115]: Waiting for error group managing segments... Dec 29 08:46:49.608469854 (569h17m49.479552889s) [noise=true , address=145.79.227.5:4221 , piece_id=THB3MJWYUU6EB4O6DUJ33V644WP247CF5N4UBEKV2OTWDELE5NRA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.608871929 (569h17m49.479954964s) [noise=true , address=145.79.227.48:4303 , piece_id=SBUUIIPD4YEEYHUHHY46UJLVD5YBMTUSAO54QCTSFPNKD5YNV6IQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.612511811 (569h17m49.483594846s) [noise=true , address=145.79.227.38:4978 , piece_id=R5TZIOYZIB3TJVAWV2JN6RLQ6IAN2ZSDU2YZVW4OIIDSTR7IP2JQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.619053035 (569h17m49.490136071s) [noise=true , address=145.79.227.22:4509 , piece_id=FGKOGHI3SFNGEEBUAC2MU2CHI7LR2A6PUQEAATZ64IYD2CEVOPQQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.632098837 (569h17m49.503181882s) [noise=true , address=145.79.227.9:4367 , piece_id=7YH3VPUEPA6CQHFJ6MZAGGSJ3KYVH623ZUPB3KLWZNMT3CIHJF6Q, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.633855652 (569h17m49.504938687s) [noise=true , address=145.79.227.52:4445 , piece_id=IEXXJ7RP5524JUHQBESOOM2CXUDCINICHIIPIRTZW2BVDOANKECQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.634974503 (569h17m49.506057548s) [noise=true , address=145.79.227.36:4517 , piece_id=UQUYFUJN27A2XFPNOTEKNUJCP4JCKDFQ7HR4U2HJBCPC4SDVEO2Q, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.635488862 (569h17m49.506571897s) [noise=true , address=145.79.227.45:4230 , piece_id=FGWGYTFTQOIGWL46UIPMRXVY27HAZBHFO6TQAXFWMK47SLWTH5GQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.636843902 (569h17m49.507926947s) [noise=true , address=145.79.227.41:4086 , piece_id=HSZKIFSMXZEYHJWPORVJI7V4VLL5Z4CQYGMFHT55GQCTPNXDG6IA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.796304019 (569h17m49.667387064s) [noise=true , address=77.90.31.10:4517 , piece_id=CUPF4TEOOKGJUXTWWJC56O7RUQBQRGGHIVIFFUI7ZX3QLLDL4ERA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.799789637 (569h17m49.670872672s) [noise=true , address=77.90.31.15:4697 , piece_id=GSYH4OXG4YUKGT7GMSMLLBPT6EACKAZGXCLQ2KE5XJR4KYEGZAWA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.892877951 (569h17m49.763960986s) [noise=true , address=45.91.116.14:4190 , piece_id=455VBRBSG6BKGVG2LXOEOSHEXREZOZD6YIOSJ2NETLQRPOSWIAPA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.904896146 (569h17m49.775979181s) [noise=true , address=45.91.116.7:4010 , piece_id=BIRGU2P5CFK2L7OWE7QXFHROVBTFQKB6QM3SW5IKF5H3RJH3UDSQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.904934258 (569h17m49.776017293s) [seg_pos={0 0}, upload=499115]: Segment reached optimal threshold of 13 pieces. Dec 29 08:46:49.905122045 (569h17m49.77620508s) [noise=true , address=45.91.116.12:4228 , piece_id=HMCUUC5ATKK6QLR47TQVZ74HGALA7UAQZ5Q42RTVLIOBIZ6Q7YUA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:37834->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:37834->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37834->45.91.116.12:4228: use of closed network connection Dec 29 08:46:49.905130813 (569h17m49.776213848s) [noise=true , address=45.91.116.12:4228 , piece_id=HMCUUC5ATKK6QLR47TQVZ74HGALA7UAQZ5Q42RTVLIOBIZ6Q7YUA, seg_pos={0 0}, upload=499115]: Long tail cancellation due to optimal threshold Dec 29 08:46:49.905234791 (569h17m49.776317826s) [noise=true , address=77.90.31.17:4773 , piece_id=LPQLAK64IRWJURQBFFNKWBEOHZND4TGRFQZSWKFEPKG5755XMGIA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 29 08:46:49.905244790 (569h17m49.776327835s) [noise=true , address=77.90.31.17:4773 , piece_id=LPQLAK64IRWJURQBFFNKWBEOHZND4TGRFQZSWKFEPKG5755XMGIA, seg_pos={0 0}, upload=499115]: Long tail cancellation due to optimal threshold Dec 29 08:46:49.905305265 (569h17m49.7763883s) [noise=true , address=77.90.31.19:4844 , piece_id=ZAECYYMVBYNVRJMKDUZOQQX75NSRP5MEC4FN65HA7OFVUTTCVL4Q, seg_pos={0 0}, upload=499115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:36966->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:36966->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36966->77.90.31.19:4844: use of closed network connection Dec 29 08:46:49.905311346 (569h17m49.776394392s) [noise=true , address=77.90.31.19:4844 , piece_id=ZAECYYMVBYNVRJMKDUZOQQX75NSRP5MEC4FN65HA7OFVUTTCVL4Q, seg_pos={0 0}, upload=499115]: Long tail cancellation due to optimal threshold Dec 29 08:46:49.905346333 (569h17m49.776429378s) [seg_pos={0 0}, upload=499115]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 08:46:49.905429671 (569h17m49.776512726s) [upload=499115]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:46:49.905436564 (569h17m49.776519609s) [upload=499115]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:46:50.203840142 (569h17m50.074923187s) [upload=499115]: Done uploading segments... Dec 29 08:46:50.203873646 (569h17m50.074956681s) [upload=499115]: Upload finished. err: <nil> Duration: 5.924480868s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.....X..........X.X..X..X.......X............X..............] duration: 3.40706602s trace: https://jaeger.dev.storj.io/trace/2f5d9cf3d6b691bb FAIL (2025-12-29 08:50:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.40709258s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X..................................X.............] duration: 35.688329583s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 08:50:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 35.688347547s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [......X..........X.X..X..X.......X............X.............] duration: 3.509339473s trace: https://jaeger.dev.storj.io/trace/2f5d9cf3d6b691bb FAIL (2025-12-29 08:50:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.509397104s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X..................................X............] duration: 22.362208331s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 08:49:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 22.362236774s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [....X.X..X...X..X...X...X...X.....XXX......X....X..XXX..X...] duration: 5.924087406s trace: https://jaeger.dev.storj.io/trace/5d1cfb69a266dd0b FAIL (2025-12-29 08:49:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 29 08:46:47.371802386 (569h17m47.242885532s) [upload=499115]: Got next segment. Inline: false Dec 29 08:46:47.371866869 (569h17m47.242949935s) [upload=499115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 08:46:47.371893961 (569h17m47.242977036s) [upload=499115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 08:46:47.619452822 (569h17m47.490535868s) [seg_pos={0 0}, upload=499115]: Begin upload segment... Dec 29 08:46:47.619498429 (569h17m47.490581464s) [seg_pos={0 0}, upload=499115]: Done begin upload segment. Dec 29 08:46:47.619545328 (569h17m47.490628363s) [noise=true , address=45.91.116.14:4190 , piece_id=455VBRBSG6BKGVG2LXOEOSHEXREZOZD6YIOSJ2NETLQRPOSWIAPA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.619604240 (569h17m47.490687286s) [noise=true , address=77.90.31.19:4844 , piece_id=ZAECYYMVBYNVRJMKDUZOQQX75NSRP5MEC4FN65HA7OFVUTTCVL4Q, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.620583435 (569h17m47.49166647s) [noise=true , address=77.90.31.10:4517 , piece_id=CUPF4TEOOKGJUXTWWJC56O7RUQBQRGGHIVIFFUI7ZX3QLLDL4ERA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.620988637 (569h17m47.492071672s) [noise=true , address=145.79.227.52:4445 , piece_id=IEXXJ7RP5524JUHQBESOOM2CXUDCINICHIIPIRTZW2BVDOANKECQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.621302364 (569h17m47.492385399s) [noise=true , address=45.91.116.12:4228 , piece_id=HMCUUC5ATKK6QLR47TQVZ74HGALA7UAQZ5Q42RTVLIOBIZ6Q7YUA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.621613196 (569h17m47.492696241s) [noise=true , address=77.90.31.17:4773 , piece_id=LPQLAK64IRWJURQBFFNKWBEOHZND4TGRFQZSWKFEPKG5755XMGIA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622000353 (569h17m47.493083398s) [noise=true , address=77.90.31.15:4697 , piece_id=GSYH4OXG4YUKGT7GMSMLLBPT6EACKAZGXCLQ2KE5XJR4KYEGZAWA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622046350 (569h17m47.493129386s) [noise=true , address=145.79.227.38:4978 , piece_id=R5TZIOYZIB3TJVAWV2JN6RLQ6IAN2ZSDU2YZVW4OIIDSTR7IP2JQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622289293 (569h17m47.493372328s) [noise=true , address=145.79.227.48:4303 , piece_id=SBUUIIPD4YEEYHUHHY46UJLVD5YBMTUSAO54QCTSFPNKD5YNV6IQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622478693 (569h17m47.493561728s) [noise=true , address=145.79.227.5:4221 , piece_id=THB3MJWYUU6EB4O6DUJ33V644WP247CF5N4UBEKV2OTWDELE5NRA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622595486 (569h17m47.493678531s) [noise=true , address=145.79.227.22:4509 , piece_id=FGKOGHI3SFNGEEBUAC2MU2CHI7LR2A6PUQEAATZ64IYD2CEVOPQQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622948328 (569h17m47.494031363s) [noise=true , address=145.79.227.45:4230 , piece_id=FGWGYTFTQOIGWL46UIPMRXVY27HAZBHFO6TQAXFWMK47SLWTH5GQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622999846 (569h17m47.494082881s) [noise=true , address=145.79.227.41:4086 , piece_id=HSZKIFSMXZEYHJWPORVJI7V4VLL5Z4CQYGMFHT55GQCTPNXDG6IA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.623421570 (569h17m47.494504615s) [noise=true , address=45.91.116.7:4010 , piece_id=BIRGU2P5CFK2L7OWE7QXFHROVBTFQKB6QM3SW5IKF5H3RJH3UDSQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.623927052 (569h17m47.495010087s) [noise=true , address=145.79.227.36:4517 , piece_id=UQUYFUJN27A2XFPNOTEKNUJCP4JCKDFQ7HR4U2HJBCPC4SDVEO2Q, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.624227705 (569h17m47.49531074s) [noise=true , address=145.79.227.9:4367 , piece_id=7YH3VPUEPA6CQHFJ6MZAGGSJ3KYVH623ZUPB3KLWZNMT3CIHJF6Q, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:49.328824315 (569h17m49.19990736s) [upload=499115]: Next returned nil segment Dec 29 08:46:49.328859142 (569h17m49.199942197s) [upload=499115]: Waiting for error group managing segments... Dec 29 08:46:49.608469854 (569h17m49.479552889s) [noise=true , address=145.79.227.5:4221 , piece_id=THB3MJWYUU6EB4O6DUJ33V644WP247CF5N4UBEKV2OTWDELE5NRA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.608871929 (569h17m49.479954964s) [noise=true , address=145.79.227.48:4303 , piece_id=SBUUIIPD4YEEYHUHHY46UJLVD5YBMTUSAO54QCTSFPNKD5YNV6IQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.612511811 (569h17m49.483594846s) [noise=true , address=145.79.227.38:4978 , piece_id=R5TZIOYZIB3TJVAWV2JN6RLQ6IAN2ZSDU2YZVW4OIIDSTR7IP2JQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.619053035 (569h17m49.490136071s) [noise=true , address=145.79.227.22:4509 , piece_id=FGKOGHI3SFNGEEBUAC2MU2CHI7LR2A6PUQEAATZ64IYD2CEVOPQQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.632098837 (569h17m49.503181882s) [noise=true , address=145.79.227.9:4367 , piece_id=7YH3VPUEPA6CQHFJ6MZAGGSJ3KYVH623ZUPB3KLWZNMT3CIHJF6Q, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.633855652 (569h17m49.504938687s) [noise=true , address=145.79.227.52:4445 , piece_id=IEXXJ7RP5524JUHQBESOOM2CXUDCINICHIIPIRTZW2BVDOANKECQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.634974503 (569h17m49.506057548s) [noise=true , address=145.79.227.36:4517 , piece_id=UQUYFUJN27A2XFPNOTEKNUJCP4JCKDFQ7HR4U2HJBCPC4SDVEO2Q, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.635488862 (569h17m49.506571897s) [noise=true , address=145.79.227.45:4230 , piece_id=FGWGYTFTQOIGWL46UIPMRXVY27HAZBHFO6TQAXFWMK47SLWTH5GQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.636843902 (569h17m49.507926947s) [noise=true , address=145.79.227.41:4086 , piece_id=HSZKIFSMXZEYHJWPORVJI7V4VLL5Z4CQYGMFHT55GQCTPNXDG6IA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.796304019 (569h17m49.667387064s) [noise=true , address=77.90.31.10:4517 , piece_id=CUPF4TEOOKGJUXTWWJC56O7RUQBQRGGHIVIFFUI7ZX3QLLDL4ERA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.799789637 (569h17m49.670872672s) [noise=true , address=77.90.31.15:4697 , piece_id=GSYH4OXG4YUKGT7GMSMLLBPT6EACKAZGXCLQ2KE5XJR4KYEGZAWA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.892877951 (569h17m49.763960986s) [noise=true , address=45.91.116.14:4190 , piece_id=455VBRBSG6BKGVG2LXOEOSHEXREZOZD6YIOSJ2NETLQRPOSWIAPA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.904896146 (569h17m49.775979181s) [noise=true , address=45.91.116.7:4010 , piece_id=BIRGU2P5CFK2L7OWE7QXFHROVBTFQKB6QM3SW5IKF5H3RJH3UDSQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.904934258 (569h17m49.776017293s) [seg_pos={0 0}, upload=499115]: Segment reached optimal threshold of 13 pieces. Dec 29 08:46:49.905122045 (569h17m49.77620508s) [noise=true , address=45.91.116.12:4228 , piece_id=HMCUUC5ATKK6QLR47TQVZ74HGALA7UAQZ5Q42RTVLIOBIZ6Q7YUA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:37834->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:37834->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37834->45.91.116.12:4228: use of closed network connection Dec 29 08:46:49.905130813 (569h17m49.776213848s) [noise=true , address=45.91.116.12:4228 , piece_id=HMCUUC5ATKK6QLR47TQVZ74HGALA7UAQZ5Q42RTVLIOBIZ6Q7YUA, seg_pos={0 0}, upload=499115]: Long tail cancellation due to optimal threshold Dec 29 08:46:49.905234791 (569h17m49.776317826s) [noise=true , address=77.90.31.17:4773 , piece_id=LPQLAK64IRWJURQBFFNKWBEOHZND4TGRFQZSWKFEPKG5755XMGIA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 29 08:46:49.905244790 (569h17m49.776327835s) [noise=true , address=77.90.31.17:4773 , piece_id=LPQLAK64IRWJURQBFFNKWBEOHZND4TGRFQZSWKFEPKG5755XMGIA, seg_pos={0 0}, upload=499115]: Long tail cancellation due to optimal threshold Dec 29 08:46:49.905305265 (569h17m49.7763883s) [noise=true , address=77.90.31.19:4844 , piece_id=ZAECYYMVBYNVRJMKDUZOQQX75NSRP5MEC4FN65HA7OFVUTTCVL4Q, seg_pos={0 0}, upload=499115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:36966->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:36966->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36966->77.90.31.19:4844: use of closed network connection Dec 29 08:46:49.905311346 (569h17m49.776394392s) [noise=true , address=77.90.31.19:4844 , piece_id=ZAECYYMVBYNVRJMKDUZOQQX75NSRP5MEC4FN65HA7OFVUTTCVL4Q, seg_pos={0 0}, upload=499115]: Long tail cancellation due to optimal threshold Dec 29 08:46:49.905346333 (569h17m49.776429378s) [seg_pos={0 0}, upload=499115]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 08:46:49.905429671 (569h17m49.776512726s) [upload=499115]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:46:49.905436564 (569h17m49.776519609s) [upload=499115]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:46:50.203840142 (569h17m50.074923187s) [upload=499115]: Done uploading segments... Dec 29 08:46:50.203873646 (569h17m50.074956681s) [upload=499115]: Upload finished. err: <nil> Duration: 5.924119115s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [X......X..........X.X..X..X.......X............X............] duration: 3.463451955s trace: https://jaeger.dev.storj.io/trace/2f5d9cf3d6b691bb FAIL (2025-12-29 08:49:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.463473958s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X..................................X...........] duration: 15.304048563s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 08:48:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 15.304078801s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X....X.X..X...X..X...X...X...X.....XXX......X....X..XXX..X..] duration: 23.652365861s trace: https://jaeger.dev.storj.io/trace/5d1cfb69a266dd0b FAIL (2025-12-29 08:48:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 29 08:46:47.371802386 (569h17m47.242885532s) [upload=499115]: Got next segment. Inline: false Dec 29 08:46:47.371866869 (569h17m47.242949935s) [upload=499115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 08:46:47.371893961 (569h17m47.242977036s) [upload=499115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 08:46:47.619452822 (569h17m47.490535868s) [seg_pos={0 0}, upload=499115]: Begin upload segment... Dec 29 08:46:47.619498429 (569h17m47.490581464s) [seg_pos={0 0}, upload=499115]: Done begin upload segment. Dec 29 08:46:47.619545328 (569h17m47.490628363s) [noise=true , address=45.91.116.14:4190 , piece_id=455VBRBSG6BKGVG2LXOEOSHEXREZOZD6YIOSJ2NETLQRPOSWIAPA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.619604240 (569h17m47.490687286s) [noise=true , address=77.90.31.19:4844 , piece_id=ZAECYYMVBYNVRJMKDUZOQQX75NSRP5MEC4FN65HA7OFVUTTCVL4Q, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.620583435 (569h17m47.49166647s) [noise=true , address=77.90.31.10:4517 , piece_id=CUPF4TEOOKGJUXTWWJC56O7RUQBQRGGHIVIFFUI7ZX3QLLDL4ERA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.620988637 (569h17m47.492071672s) [noise=true , address=145.79.227.52:4445 , piece_id=IEXXJ7RP5524JUHQBESOOM2CXUDCINICHIIPIRTZW2BVDOANKECQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.621302364 (569h17m47.492385399s) [noise=true , address=45.91.116.12:4228 , piece_id=HMCUUC5ATKK6QLR47TQVZ74HGALA7UAQZ5Q42RTVLIOBIZ6Q7YUA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.621613196 (569h17m47.492696241s) [noise=true , address=77.90.31.17:4773 , piece_id=LPQLAK64IRWJURQBFFNKWBEOHZND4TGRFQZSWKFEPKG5755XMGIA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622000353 (569h17m47.493083398s) [noise=true , address=77.90.31.15:4697 , piece_id=GSYH4OXG4YUKGT7GMSMLLBPT6EACKAZGXCLQ2KE5XJR4KYEGZAWA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622046350 (569h17m47.493129386s) [noise=true , address=145.79.227.38:4978 , piece_id=R5TZIOYZIB3TJVAWV2JN6RLQ6IAN2ZSDU2YZVW4OIIDSTR7IP2JQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622289293 (569h17m47.493372328s) [noise=true , address=145.79.227.48:4303 , piece_id=SBUUIIPD4YEEYHUHHY46UJLVD5YBMTUSAO54QCTSFPNKD5YNV6IQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622478693 (569h17m47.493561728s) [noise=true , address=145.79.227.5:4221 , piece_id=THB3MJWYUU6EB4O6DUJ33V644WP247CF5N4UBEKV2OTWDELE5NRA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622595486 (569h17m47.493678531s) [noise=true , address=145.79.227.22:4509 , piece_id=FGKOGHI3SFNGEEBUAC2MU2CHI7LR2A6PUQEAATZ64IYD2CEVOPQQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622948328 (569h17m47.494031363s) [noise=true , address=145.79.227.45:4230 , piece_id=FGWGYTFTQOIGWL46UIPMRXVY27HAZBHFO6TQAXFWMK47SLWTH5GQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622999846 (569h17m47.494082881s) [noise=true , address=145.79.227.41:4086 , piece_id=HSZKIFSMXZEYHJWPORVJI7V4VLL5Z4CQYGMFHT55GQCTPNXDG6IA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.623421570 (569h17m47.494504615s) [noise=true , address=45.91.116.7:4010 , piece_id=BIRGU2P5CFK2L7OWE7QXFHROVBTFQKB6QM3SW5IKF5H3RJH3UDSQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.623927052 (569h17m47.495010087s) [noise=true , address=145.79.227.36:4517 , piece_id=UQUYFUJN27A2XFPNOTEKNUJCP4JCKDFQ7HR4U2HJBCPC4SDVEO2Q, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.624227705 (569h17m47.49531074s) [noise=true , address=145.79.227.9:4367 , piece_id=7YH3VPUEPA6CQHFJ6MZAGGSJ3KYVH623ZUPB3KLWZNMT3CIHJF6Q, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:49.328824315 (569h17m49.19990736s) [upload=499115]: Next returned nil segment Dec 29 08:46:49.328859142 (569h17m49.199942197s) [upload=499115]: Waiting for error group managing segments... Dec 29 08:46:49.608469854 (569h17m49.479552889s) [noise=true , address=145.79.227.5:4221 , piece_id=THB3MJWYUU6EB4O6DUJ33V644WP247CF5N4UBEKV2OTWDELE5NRA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.608871929 (569h17m49.479954964s) [noise=true , address=145.79.227.48:4303 , piece_id=SBUUIIPD4YEEYHUHHY46UJLVD5YBMTUSAO54QCTSFPNKD5YNV6IQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.612511811 (569h17m49.483594846s) [noise=true , address=145.79.227.38:4978 , piece_id=R5TZIOYZIB3TJVAWV2JN6RLQ6IAN2ZSDU2YZVW4OIIDSTR7IP2JQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.619053035 (569h17m49.490136071s) [noise=true , address=145.79.227.22:4509 , piece_id=FGKOGHI3SFNGEEBUAC2MU2CHI7LR2A6PUQEAATZ64IYD2CEVOPQQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.632098837 (569h17m49.503181882s) [noise=true , address=145.79.227.9:4367 , piece_id=7YH3VPUEPA6CQHFJ6MZAGGSJ3KYVH623ZUPB3KLWZNMT3CIHJF6Q, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.633855652 (569h17m49.504938687s) [noise=true , address=145.79.227.52:4445 , piece_id=IEXXJ7RP5524JUHQBESOOM2CXUDCINICHIIPIRTZW2BVDOANKECQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.634974503 (569h17m49.506057548s) [noise=true , address=145.79.227.36:4517 , piece_id=UQUYFUJN27A2XFPNOTEKNUJCP4JCKDFQ7HR4U2HJBCPC4SDVEO2Q, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.635488862 (569h17m49.506571897s) [noise=true , address=145.79.227.45:4230 , piece_id=FGWGYTFTQOIGWL46UIPMRXVY27HAZBHFO6TQAXFWMK47SLWTH5GQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.636843902 (569h17m49.507926947s) [noise=true , address=145.79.227.41:4086 , piece_id=HSZKIFSMXZEYHJWPORVJI7V4VLL5Z4CQYGMFHT55GQCTPNXDG6IA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.796304019 (569h17m49.667387064s) [noise=true , address=77.90.31.10:4517 , piece_id=CUPF4TEOOKGJUXTWWJC56O7RUQBQRGGHIVIFFUI7ZX3QLLDL4ERA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.799789637 (569h17m49.670872672s) [noise=true , address=77.90.31.15:4697 , piece_id=GSYH4OXG4YUKGT7GMSMLLBPT6EACKAZGXCLQ2KE5XJR4KYEGZAWA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.892877951 (569h17m49.763960986s) [noise=true , address=45.91.116.14:4190 , piece_id=455VBRBSG6BKGVG2LXOEOSHEXREZOZD6YIOSJ2NETLQRPOSWIAPA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.904896146 (569h17m49.775979181s) [noise=true , address=45.91.116.7:4010 , piece_id=BIRGU2P5CFK2L7OWE7QXFHROVBTFQKB6QM3SW5IKF5H3RJH3UDSQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.904934258 (569h17m49.776017293s) [seg_pos={0 0}, upload=499115]: Segment reached optimal threshold of 13 pieces. Dec 29 08:46:49.905122045 (569h17m49.77620508s) [noise=true , address=45.91.116.12:4228 , piece_id=HMCUUC5ATKK6QLR47TQVZ74HGALA7UAQZ5Q42RTVLIOBIZ6Q7YUA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:37834->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:37834->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37834->45.91.116.12:4228: use of closed network connection Dec 29 08:46:49.905130813 (569h17m49.776213848s) [noise=true , address=45.91.116.12:4228 , piece_id=HMCUUC5ATKK6QLR47TQVZ74HGALA7UAQZ5Q42RTVLIOBIZ6Q7YUA, seg_pos={0 0}, upload=499115]: Long tail cancellation due to optimal threshold Dec 29 08:46:49.905234791 (569h17m49.776317826s) [noise=true , address=77.90.31.17:4773 , piece_id=LPQLAK64IRWJURQBFFNKWBEOHZND4TGRFQZSWKFEPKG5755XMGIA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 29 08:46:49.905244790 (569h17m49.776327835s) [noise=true , address=77.90.31.17:4773 , piece_id=LPQLAK64IRWJURQBFFNKWBEOHZND4TGRFQZSWKFEPKG5755XMGIA, seg_pos={0 0}, upload=499115]: Long tail cancellation due to optimal threshold Dec 29 08:46:49.905305265 (569h17m49.7763883s) [noise=true , address=77.90.31.19:4844 , piece_id=ZAECYYMVBYNVRJMKDUZOQQX75NSRP5MEC4FN65HA7OFVUTTCVL4Q, seg_pos={0 0}, upload=499115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:36966->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:36966->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36966->77.90.31.19:4844: use of closed network connection Dec 29 08:46:49.905311346 (569h17m49.776394392s) [noise=true , address=77.90.31.19:4844 , piece_id=ZAECYYMVBYNVRJMKDUZOQQX75NSRP5MEC4FN65HA7OFVUTTCVL4Q, seg_pos={0 0}, upload=499115]: Long tail cancellation due to optimal threshold Dec 29 08:46:49.905346333 (569h17m49.776429378s) [seg_pos={0 0}, upload=499115]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 08:46:49.905429671 (569h17m49.776512726s) [upload=499115]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:46:49.905436564 (569h17m49.776519609s) [upload=499115]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:46:50.203840142 (569h17m50.074923187s) [upload=499115]: Done uploading segments... Dec 29 08:46:50.203873646 (569h17m50.074956681s) [upload=499115]: Upload finished. err: <nil> Duration: 23.652396578s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.X......X..........X.X..X..X.......X............X...........] duration: 3.619416649s trace: https://jaeger.dev.storj.io/trace/2f5d9cf3d6b691bb FAIL (2025-12-29 08:48:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.619448019s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X..................................X..........] duration: 36.98449183s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 08:48:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 36.984533701s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XX....X.X..X...X..X...X...X...X.....XXX......X....X..XXX..X.] duration: 6.122840493s trace: https://jaeger.dev.storj.io/trace/5d1cfb69a266dd0b FAIL (2025-12-29 08:47:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 29 08:46:47.371802386 (569h17m47.242885532s) [upload=499115]: Got next segment. Inline: false Dec 29 08:46:47.371866869 (569h17m47.242949935s) [upload=499115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 08:46:47.371893961 (569h17m47.242977036s) [upload=499115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 08:46:47.619452822 (569h17m47.490535868s) [seg_pos={0 0}, upload=499115]: Begin upload segment... Dec 29 08:46:47.619498429 (569h17m47.490581464s) [seg_pos={0 0}, upload=499115]: Done begin upload segment. Dec 29 08:46:47.619545328 (569h17m47.490628363s) [noise=true , address=45.91.116.14:4190 , piece_id=455VBRBSG6BKGVG2LXOEOSHEXREZOZD6YIOSJ2NETLQRPOSWIAPA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.619604240 (569h17m47.490687286s) [noise=true , address=77.90.31.19:4844 , piece_id=ZAECYYMVBYNVRJMKDUZOQQX75NSRP5MEC4FN65HA7OFVUTTCVL4Q, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.620583435 (569h17m47.49166647s) [noise=true , address=77.90.31.10:4517 , piece_id=CUPF4TEOOKGJUXTWWJC56O7RUQBQRGGHIVIFFUI7ZX3QLLDL4ERA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.620988637 (569h17m47.492071672s) [noise=true , address=145.79.227.52:4445 , piece_id=IEXXJ7RP5524JUHQBESOOM2CXUDCINICHIIPIRTZW2BVDOANKECQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.621302364 (569h17m47.492385399s) [noise=true , address=45.91.116.12:4228 , piece_id=HMCUUC5ATKK6QLR47TQVZ74HGALA7UAQZ5Q42RTVLIOBIZ6Q7YUA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.621613196 (569h17m47.492696241s) [noise=true , address=77.90.31.17:4773 , piece_id=LPQLAK64IRWJURQBFFNKWBEOHZND4TGRFQZSWKFEPKG5755XMGIA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622000353 (569h17m47.493083398s) [noise=true , address=77.90.31.15:4697 , piece_id=GSYH4OXG4YUKGT7GMSMLLBPT6EACKAZGXCLQ2KE5XJR4KYEGZAWA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622046350 (569h17m47.493129386s) [noise=true , address=145.79.227.38:4978 , piece_id=R5TZIOYZIB3TJVAWV2JN6RLQ6IAN2ZSDU2YZVW4OIIDSTR7IP2JQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622289293 (569h17m47.493372328s) [noise=true , address=145.79.227.48:4303 , piece_id=SBUUIIPD4YEEYHUHHY46UJLVD5YBMTUSAO54QCTSFPNKD5YNV6IQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622478693 (569h17m47.493561728s) [noise=true , address=145.79.227.5:4221 , piece_id=THB3MJWYUU6EB4O6DUJ33V644WP247CF5N4UBEKV2OTWDELE5NRA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622595486 (569h17m47.493678531s) [noise=true , address=145.79.227.22:4509 , piece_id=FGKOGHI3SFNGEEBUAC2MU2CHI7LR2A6PUQEAATZ64IYD2CEVOPQQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622948328 (569h17m47.494031363s) [noise=true , address=145.79.227.45:4230 , piece_id=FGWGYTFTQOIGWL46UIPMRXVY27HAZBHFO6TQAXFWMK47SLWTH5GQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622999846 (569h17m47.494082881s) [noise=true , address=145.79.227.41:4086 , piece_id=HSZKIFSMXZEYHJWPORVJI7V4VLL5Z4CQYGMFHT55GQCTPNXDG6IA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.623421570 (569h17m47.494504615s) [noise=true , address=45.91.116.7:4010 , piece_id=BIRGU2P5CFK2L7OWE7QXFHROVBTFQKB6QM3SW5IKF5H3RJH3UDSQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.623927052 (569h17m47.495010087s) [noise=true , address=145.79.227.36:4517 , piece_id=UQUYFUJN27A2XFPNOTEKNUJCP4JCKDFQ7HR4U2HJBCPC4SDVEO2Q, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.624227705 (569h17m47.49531074s) [noise=true , address=145.79.227.9:4367 , piece_id=7YH3VPUEPA6CQHFJ6MZAGGSJ3KYVH623ZUPB3KLWZNMT3CIHJF6Q, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:49.328824315 (569h17m49.19990736s) [upload=499115]: Next returned nil segment Dec 29 08:46:49.328859142 (569h17m49.199942197s) [upload=499115]: Waiting for error group managing segments... Dec 29 08:46:49.608469854 (569h17m49.479552889s) [noise=true , address=145.79.227.5:4221 , piece_id=THB3MJWYUU6EB4O6DUJ33V644WP247CF5N4UBEKV2OTWDELE5NRA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.608871929 (569h17m49.479954964s) [noise=true , address=145.79.227.48:4303 , piece_id=SBUUIIPD4YEEYHUHHY46UJLVD5YBMTUSAO54QCTSFPNKD5YNV6IQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.612511811 (569h17m49.483594846s) [noise=true , address=145.79.227.38:4978 , piece_id=R5TZIOYZIB3TJVAWV2JN6RLQ6IAN2ZSDU2YZVW4OIIDSTR7IP2JQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.619053035 (569h17m49.490136071s) [noise=true , address=145.79.227.22:4509 , piece_id=FGKOGHI3SFNGEEBUAC2MU2CHI7LR2A6PUQEAATZ64IYD2CEVOPQQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.632098837 (569h17m49.503181882s) [noise=true , address=145.79.227.9:4367 , piece_id=7YH3VPUEPA6CQHFJ6MZAGGSJ3KYVH623ZUPB3KLWZNMT3CIHJF6Q, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.633855652 (569h17m49.504938687s) [noise=true , address=145.79.227.52:4445 , piece_id=IEXXJ7RP5524JUHQBESOOM2CXUDCINICHIIPIRTZW2BVDOANKECQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.634974503 (569h17m49.506057548s) [noise=true , address=145.79.227.36:4517 , piece_id=UQUYFUJN27A2XFPNOTEKNUJCP4JCKDFQ7HR4U2HJBCPC4SDVEO2Q, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.635488862 (569h17m49.506571897s) [noise=true , address=145.79.227.45:4230 , piece_id=FGWGYTFTQOIGWL46UIPMRXVY27HAZBHFO6TQAXFWMK47SLWTH5GQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.636843902 (569h17m49.507926947s) [noise=true , address=145.79.227.41:4086 , piece_id=HSZKIFSMXZEYHJWPORVJI7V4VLL5Z4CQYGMFHT55GQCTPNXDG6IA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.796304019 (569h17m49.667387064s) [noise=true , address=77.90.31.10:4517 , piece_id=CUPF4TEOOKGJUXTWWJC56O7RUQBQRGGHIVIFFUI7ZX3QLLDL4ERA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.799789637 (569h17m49.670872672s) [noise=true , address=77.90.31.15:4697 , piece_id=GSYH4OXG4YUKGT7GMSMLLBPT6EACKAZGXCLQ2KE5XJR4KYEGZAWA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.892877951 (569h17m49.763960986s) [noise=true , address=45.91.116.14:4190 , piece_id=455VBRBSG6BKGVG2LXOEOSHEXREZOZD6YIOSJ2NETLQRPOSWIAPA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.904896146 (569h17m49.775979181s) [noise=true , address=45.91.116.7:4010 , piece_id=BIRGU2P5CFK2L7OWE7QXFHROVBTFQKB6QM3SW5IKF5H3RJH3UDSQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.904934258 (569h17m49.776017293s) [seg_pos={0 0}, upload=499115]: Segment reached optimal threshold of 13 pieces. Dec 29 08:46:49.905122045 (569h17m49.77620508s) [noise=true , address=45.91.116.12:4228 , piece_id=HMCUUC5ATKK6QLR47TQVZ74HGALA7UAQZ5Q42RTVLIOBIZ6Q7YUA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:37834->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:37834->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37834->45.91.116.12:4228: use of closed network connection Dec 29 08:46:49.905130813 (569h17m49.776213848s) [noise=true , address=45.91.116.12:4228 , piece_id=HMCUUC5ATKK6QLR47TQVZ74HGALA7UAQZ5Q42RTVLIOBIZ6Q7YUA, seg_pos={0 0}, upload=499115]: Long tail cancellation due to optimal threshold Dec 29 08:46:49.905234791 (569h17m49.776317826s) [noise=true , address=77.90.31.17:4773 , piece_id=LPQLAK64IRWJURQBFFNKWBEOHZND4TGRFQZSWKFEPKG5755XMGIA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 29 08:46:49.905244790 (569h17m49.776327835s) [noise=true , address=77.90.31.17:4773 , piece_id=LPQLAK64IRWJURQBFFNKWBEOHZND4TGRFQZSWKFEPKG5755XMGIA, seg_pos={0 0}, upload=499115]: Long tail cancellation due to optimal threshold Dec 29 08:46:49.905305265 (569h17m49.7763883s) [noise=true , address=77.90.31.19:4844 , piece_id=ZAECYYMVBYNVRJMKDUZOQQX75NSRP5MEC4FN65HA7OFVUTTCVL4Q, seg_pos={0 0}, upload=499115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:36966->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:36966->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36966->77.90.31.19:4844: use of closed network connection Dec 29 08:46:49.905311346 (569h17m49.776394392s) [noise=true , address=77.90.31.19:4844 , piece_id=ZAECYYMVBYNVRJMKDUZOQQX75NSRP5MEC4FN65HA7OFVUTTCVL4Q, seg_pos={0 0}, upload=499115]: Long tail cancellation due to optimal threshold Dec 29 08:46:49.905346333 (569h17m49.776429378s) [seg_pos={0 0}, upload=499115]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 08:46:49.905429671 (569h17m49.776512726s) [upload=499115]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:46:49.905436564 (569h17m49.776519609s) [upload=499115]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:46:50.203840142 (569h17m50.074923187s) [upload=499115]: Done uploading segments... Dec 29 08:46:50.203873646 (569h17m50.074956681s) [upload=499115]: Upload finished. err: <nil> Duration: 6.122867263s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.XX....X.X..X...X..X...X...X...X.....XXX......X....X..XXX..X] duration: 30.003358834s trace: https://jaeger.dev.storj.io/trace/5d1cfb69a266dd0b FAIL (2025-12-29 08:47:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 29 08:46:47.371802386 (569h17m47.242885532s) [upload=499115]: Got next segment. Inline: false Dec 29 08:46:47.371866869 (569h17m47.242949935s) [upload=499115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 08:46:47.371893961 (569h17m47.242977036s) [upload=499115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 08:46:47.619452822 (569h17m47.490535868s) [seg_pos={0 0}, upload=499115]: Begin upload segment... Dec 29 08:46:47.619498429 (569h17m47.490581464s) [seg_pos={0 0}, upload=499115]: Done begin upload segment. Dec 29 08:46:47.619545328 (569h17m47.490628363s) [noise=true , address=45.91.116.14:4190 , piece_id=455VBRBSG6BKGVG2LXOEOSHEXREZOZD6YIOSJ2NETLQRPOSWIAPA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.619604240 (569h17m47.490687286s) [noise=true , address=77.90.31.19:4844 , piece_id=ZAECYYMVBYNVRJMKDUZOQQX75NSRP5MEC4FN65HA7OFVUTTCVL4Q, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.620583435 (569h17m47.49166647s) [noise=true , address=77.90.31.10:4517 , piece_id=CUPF4TEOOKGJUXTWWJC56O7RUQBQRGGHIVIFFUI7ZX3QLLDL4ERA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.620988637 (569h17m47.492071672s) [noise=true , address=145.79.227.52:4445 , piece_id=IEXXJ7RP5524JUHQBESOOM2CXUDCINICHIIPIRTZW2BVDOANKECQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.621302364 (569h17m47.492385399s) [noise=true , address=45.91.116.12:4228 , piece_id=HMCUUC5ATKK6QLR47TQVZ74HGALA7UAQZ5Q42RTVLIOBIZ6Q7YUA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.621613196 (569h17m47.492696241s) [noise=true , address=77.90.31.17:4773 , piece_id=LPQLAK64IRWJURQBFFNKWBEOHZND4TGRFQZSWKFEPKG5755XMGIA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622000353 (569h17m47.493083398s) [noise=true , address=77.90.31.15:4697 , piece_id=GSYH4OXG4YUKGT7GMSMLLBPT6EACKAZGXCLQ2KE5XJR4KYEGZAWA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622046350 (569h17m47.493129386s) [noise=true , address=145.79.227.38:4978 , piece_id=R5TZIOYZIB3TJVAWV2JN6RLQ6IAN2ZSDU2YZVW4OIIDSTR7IP2JQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622289293 (569h17m47.493372328s) [noise=true , address=145.79.227.48:4303 , piece_id=SBUUIIPD4YEEYHUHHY46UJLVD5YBMTUSAO54QCTSFPNKD5YNV6IQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622478693 (569h17m47.493561728s) [noise=true , address=145.79.227.5:4221 , piece_id=THB3MJWYUU6EB4O6DUJ33V644WP247CF5N4UBEKV2OTWDELE5NRA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622595486 (569h17m47.493678531s) [noise=true , address=145.79.227.22:4509 , piece_id=FGKOGHI3SFNGEEBUAC2MU2CHI7LR2A6PUQEAATZ64IYD2CEVOPQQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622948328 (569h17m47.494031363s) [noise=true , address=145.79.227.45:4230 , piece_id=FGWGYTFTQOIGWL46UIPMRXVY27HAZBHFO6TQAXFWMK47SLWTH5GQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.622999846 (569h17m47.494082881s) [noise=true , address=145.79.227.41:4086 , piece_id=HSZKIFSMXZEYHJWPORVJI7V4VLL5Z4CQYGMFHT55GQCTPNXDG6IA, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.623421570 (569h17m47.494504615s) [noise=true , address=45.91.116.7:4010 , piece_id=BIRGU2P5CFK2L7OWE7QXFHROVBTFQKB6QM3SW5IKF5H3RJH3UDSQ, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.623927052 (569h17m47.495010087s) [noise=true , address=145.79.227.36:4517 , piece_id=UQUYFUJN27A2XFPNOTEKNUJCP4JCKDFQ7HR4U2HJBCPC4SDVEO2Q, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:47.624227705 (569h17m47.49531074s) [noise=true , address=145.79.227.9:4367 , piece_id=7YH3VPUEPA6CQHFJ6MZAGGSJ3KYVH623ZUPB3KLWZNMT3CIHJF6Q, seg_pos={0 0}, upload=499115]: Uploading piece... Dec 29 08:46:49.328824315 (569h17m49.19990736s) [upload=499115]: Next returned nil segment Dec 29 08:46:49.328859142 (569h17m49.199942197s) [upload=499115]: Waiting for error group managing segments... Dec 29 08:46:49.608469854 (569h17m49.479552889s) [noise=true , address=145.79.227.5:4221 , piece_id=THB3MJWYUU6EB4O6DUJ33V644WP247CF5N4UBEKV2OTWDELE5NRA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.608871929 (569h17m49.479954964s) [noise=true , address=145.79.227.48:4303 , piece_id=SBUUIIPD4YEEYHUHHY46UJLVD5YBMTUSAO54QCTSFPNKD5YNV6IQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.612511811 (569h17m49.483594846s) [noise=true , address=145.79.227.38:4978 , piece_id=R5TZIOYZIB3TJVAWV2JN6RLQ6IAN2ZSDU2YZVW4OIIDSTR7IP2JQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.619053035 (569h17m49.490136071s) [noise=true , address=145.79.227.22:4509 , piece_id=FGKOGHI3SFNGEEBUAC2MU2CHI7LR2A6PUQEAATZ64IYD2CEVOPQQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.632098837 (569h17m49.503181882s) [noise=true , address=145.79.227.9:4367 , piece_id=7YH3VPUEPA6CQHFJ6MZAGGSJ3KYVH623ZUPB3KLWZNMT3CIHJF6Q, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.633855652 (569h17m49.504938687s) [noise=true , address=145.79.227.52:4445 , piece_id=IEXXJ7RP5524JUHQBESOOM2CXUDCINICHIIPIRTZW2BVDOANKECQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.634974503 (569h17m49.506057548s) [noise=true , address=145.79.227.36:4517 , piece_id=UQUYFUJN27A2XFPNOTEKNUJCP4JCKDFQ7HR4U2HJBCPC4SDVEO2Q, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.635488862 (569h17m49.506571897s) [noise=true , address=145.79.227.45:4230 , piece_id=FGWGYTFTQOIGWL46UIPMRXVY27HAZBHFO6TQAXFWMK47SLWTH5GQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.636843902 (569h17m49.507926947s) [noise=true , address=145.79.227.41:4086 , piece_id=HSZKIFSMXZEYHJWPORVJI7V4VLL5Z4CQYGMFHT55GQCTPNXDG6IA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.796304019 (569h17m49.667387064s) [noise=true , address=77.90.31.10:4517 , piece_id=CUPF4TEOOKGJUXTWWJC56O7RUQBQRGGHIVIFFUI7ZX3QLLDL4ERA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.799789637 (569h17m49.670872672s) [noise=true , address=77.90.31.15:4697 , piece_id=GSYH4OXG4YUKGT7GMSMLLBPT6EACKAZGXCLQ2KE5XJR4KYEGZAWA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.892877951 (569h17m49.763960986s) [noise=true , address=45.91.116.14:4190 , piece_id=455VBRBSG6BKGVG2LXOEOSHEXREZOZD6YIOSJ2NETLQRPOSWIAPA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.904896146 (569h17m49.775979181s) [noise=true , address=45.91.116.7:4010 , piece_id=BIRGU2P5CFK2L7OWE7QXFHROVBTFQKB6QM3SW5IKF5H3RJH3UDSQ, seg_pos={0 0}, upload=499115]: Done uploading piece. err: <nil> Dec 29 08:46:49.904934258 (569h17m49.776017293s) [seg_pos={0 0}, upload=499115]: Segment reached optimal threshold of 13 pieces. Dec 29 08:46:49.905122045 (569h17m49.77620508s) [noise=true , address=45.91.116.12:4228 , piece_id=HMCUUC5ATKK6QLR47TQVZ74HGALA7UAQZ5Q42RTVLIOBIZ6Q7YUA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:37834->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:37834->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37834->45.91.116.12:4228: use of closed network connection Dec 29 08:46:49.905130813 (569h17m49.776213848s) [noise=true , address=45.91.116.12:4228 , piece_id=HMCUUC5ATKK6QLR47TQVZ74HGALA7UAQZ5Q42RTVLIOBIZ6Q7YUA, seg_pos={0 0}, upload=499115]: Long tail cancellation due to optimal threshold Dec 29 08:46:49.905234791 (569h17m49.776317826s) [noise=true , address=77.90.31.17:4773 , piece_id=LPQLAK64IRWJURQBFFNKWBEOHZND4TGRFQZSWKFEPKG5755XMGIA, seg_pos={0 0}, upload=499115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 29 08:46:49.905244790 (569h17m49.776327835s) [noise=true , address=77.90.31.17:4773 , piece_id=LPQLAK64IRWJURQBFFNKWBEOHZND4TGRFQZSWKFEPKG5755XMGIA, seg_pos={0 0}, upload=499115]: Long tail cancellation due to optimal threshold Dec 29 08:46:49.905305265 (569h17m49.7763883s) [noise=true , address=77.90.31.19:4844 , piece_id=ZAECYYMVBYNVRJMKDUZOQQX75NSRP5MEC4FN65HA7OFVUTTCVL4Q, seg_pos={0 0}, upload=499115]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:36966->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:36966->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36966->77.90.31.19:4844: use of closed network connection Dec 29 08:46:49.905311346 (569h17m49.776394392s) [noise=true , address=77.90.31.19:4844 , piece_id=ZAECYYMVBYNVRJMKDUZOQQX75NSRP5MEC4FN65HA7OFVUTTCVL4Q, seg_pos={0 0}, upload=499115]: Long tail cancellation due to optimal threshold Dec 29 08:46:49.905346333 (569h17m49.776429378s) [seg_pos={0 0}, upload=499115]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 08:46:49.905429671 (569h17m49.776512726s) [upload=499115]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:46:49.905436564 (569h17m49.776519609s) [upload=499115]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:46:50.203840142 (569h17m50.074923187s) [upload=499115]: Done uploading segments... Dec 29 08:46:50.203873646 (569h17m50.074956681s) [upload=499115]: Upload finished. err: <nil> Duration: 30.003381387s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..X......X..........X.X..X..X.......X............X..........] duration: 3.52043887s trace: https://jaeger.dev.storj.io/trace/2f5d9cf3d6b691bb FAIL (2025-12-29 08:47:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.520600938s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X..................................X.........] duration: 15.166938152s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 08:47:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 15.166975544s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [X..X......X..........X.X..X..X.......X............X.........] duration: 3.802645182s trace: https://jaeger.dev.storj.io/trace/2f5d9cf3d6b691bb FAIL (2025-12-29 08:46:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.802670972s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.X..X......X..........X.X..X..X.......X............X........] duration: 3.587995548s trace: https://jaeger.dev.storj.io/trace/2f5d9cf3d6b691bb FAIL (2025-12-29 08:46:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.588020316s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X..................................X........] duration: 15.352089465s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 08:45:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 15.35211859s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..XX....X.X..X...X..X...X...X...X.....XXX......X....X..XXX..] duration: 5.686139024s trace: https://jaeger.dev.storj.io/trace/6e8570c9cb7a8654 FAIL (2025-12-29 08:45:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: QIPE6ZI7EZ3PEC7STAVCXGQZ5VTGSB7BUYT6GYBYQBGEBD7XGVYQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: URTDVRL77WJTQ3GFYVWNSNTRTGKG7HGEBH7UCU7MJFZVOXQUYX6A) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: GAELOORMH7VZWU3N6JW4DT2JHQQZQHI2FCF5OHSICND45OAKNMTA) context deadline exceeded ======LOGS====== Dec 29 08:43:56.699530751 (569h14m56.570613786s) [upload=499074]: Starting upload Dec 29 08:43:56.699603459 (569h14m56.570686494s) [upload=499074]: Done starting upload Dec 29 08:43:56.699706125 (569h14m56.57078916s) [upload=499074]: Uploading segments... Dec 29 08:43:56.699824461 (569h14m56.570907496s) [upload=499074]: Got next segment. Inline: false Dec 29 08:43:56.699922538 (569h14m56.571005573s) [upload=499074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 08:43:56.699931024 (569h14m56.571014059s) [upload=499074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 08:43:56.973375414 (569h14m56.844458459s) [seg_pos={0 0}, upload=499074]: Begin upload segment... Dec 29 08:43:56.973432723 (569h14m56.844515758s) [seg_pos={0 0}, upload=499074]: Done begin upload segment. Dec 29 08:43:56.973474783 (569h14m56.844557828s) [noise=true , address=145.79.227.48:4301 , piece_id=EHRCCKPBAS3ZMRDERMTRI536RXN4EAHZ5Y2V67GAJUHUWHWDQX3Q, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.973493448 (569h14m56.844576483s) [noise=true , address=145.79.227.4:4185 , piece_id=M3TVHGCW4K2CN5CYM7XAPBOKOZJ77AON4HKZ4F726Y5YLSEIXWBQ, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.973595763 (569h14m56.844678808s) [noise=true , address=145.79.227.12:4765 , piece_id=VQXSWQUAJQPBKBGJFOL4UHWSRF7L6VBKGO65PDHDIAMQJVATC4RQ, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.974421145 (569h14m56.84550419s) [noise=true , address=145.79.227.11:4729 , piece_id=SFOJPHTKYY3UAAIZLG3EHSYEJPUY7CXBS7YAIANXDK3LAFWEIALA, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.974712920 (569h14m56.845795955s) [noise=true , address=45.91.116.9:4261 , piece_id=QIPE6ZI7EZ3PEC7STAVCXGQZ5VTGSB7BUYT6GYBYQBGEBD7XGVYQ, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.974957005 (569h14m56.84604005s) [noise=true , address=145.79.227.30:4762 , piece_id=2OX6ZRC6JZACGGDXIZTIYHWY47W5OHUYNIR5QRIIPEGCKDFTDKIA, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.975230316 (569h14m56.846313351s) [noise=true , address=45.91.116.12:4226 , piece_id=YP7OX6N4HI5DBNSWKTO3Y5WBEZJ2OZW6NDJC6I4TT267XUAXBVEQ, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.975669773 (569h14m56.846752818s) [noise=true , address=45.91.116.14:4191 , piece_id=VLKKHXI2N6TUR3WST6QIXQTA6QWK5NPFMWTSO73YHNRI3VWJ25XA, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.975776065 (569h14m56.84685911s) [noise=true , address=77.90.31.10:4517 , piece_id=T6IGCT265KAHVZOKK3SOROS6DWMLI6ZZVL35LBXGDUCPRBZFQ3YQ, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.976343445 (569h14m56.84742649s) [noise=true , address=145.79.227.37:4565 , piece_id=UUNLR2KIA2OB7VK5ETJNMVWMQ6QTOKRUCIIB3XXR4XNNW4DK6KRQ, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.976406325 (569h14m56.84748936s) [noise=true , address=45.91.116.10:4081 , piece_id=URTDVRL77WJTQ3GFYVWNSNTRTGKG7HGEBH7UCU7MJFZVOXQUYX6A, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.976830953 (569h14m56.847913988s) [noise=true , address=45.91.116.13:4154 , piece_id=NFHUV4S3WTCATYYXELPZFRKLGIG7CXGYIOVPHVWLFDWTDYNGSVTQ, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.977407381 (569h14m56.848490417s) [noise=true , address=145.79.227.13:4803 , piece_id=PIWP4ZFYI5JK2KOYEYFX6CY6VVYWKUAD2JRMUPU7KMGSGN4HNREQ, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.977933213 (569h14m56.849016248s) [noise=true , address=145.79.227.45:4231 , piece_id=E6NESNDU7TIZ66ZVAUSYMKYHSZAWOCN6PHUJDIFRYV4ROUOI44QA, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.978581717 (569h14m56.849664752s) [noise=true , address=145.79.227.14:4838 , piece_id=VUCUTQUPDYUQOPCCSZJ52KSYZKITWQ5VV6W4KUDEH6OJXNLR34XA, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.979101036 (569h14m56.850184081s) [noise=true , address=77.90.31.15:4697 , piece_id=GAELOORMH7VZWU3N6JW4DT2JHQQZQHI2FCF5OHSICND45OAKNMTA, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:58.668089757 (569h14m58.539172792s) [upload=499074]: Next returned nil segment Dec 29 08:43:58.668107570 (569h14m58.539190615s) [upload=499074]: Waiting for error group managing segments... Dec 29 08:43:58.958856674 (569h14m58.82993972s) [noise=true , address=145.79.227.14:4838 , piece_id=VUCUTQUPDYUQOPCCSZJ52KSYZKITWQ5VV6W4KUDEH6OJXNLR34XA, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:58.960420802 (569h14m58.831503837s) [noise=true , address=145.79.227.4:4185 , piece_id=M3TVHGCW4K2CN5CYM7XAPBOKOZJ77AON4HKZ4F726Y5YLSEIXWBQ, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:58.967931053 (569h14m58.839014098s) [noise=true , address=145.79.227.11:4729 , piece_id=SFOJPHTKYY3UAAIZLG3EHSYEJPUY7CXBS7YAIANXDK3LAFWEIALA, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:58.970437576 (569h14m58.841520611s) [noise=true , address=145.79.227.13:4803 , piece_id=PIWP4ZFYI5JK2KOYEYFX6CY6VVYWKUAD2JRMUPU7KMGSGN4HNREQ, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:58.980239059 (569h14m58.851322104s) [noise=true , address=145.79.227.12:4765 , piece_id=VQXSWQUAJQPBKBGJFOL4UHWSRF7L6VBKGO65PDHDIAMQJVATC4RQ, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:58.980629754 (569h14m58.851712789s) [noise=true , address=145.79.227.45:4231 , piece_id=E6NESNDU7TIZ66ZVAUSYMKYHSZAWOCN6PHUJDIFRYV4ROUOI44QA, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:58.990476994 (569h14m58.861560039s) [noise=true , address=145.79.227.48:4301 , piece_id=EHRCCKPBAS3ZMRDERMTRI536RXN4EAHZ5Y2V67GAJUHUWHWDQX3Q, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:58.999989858 (569h14m58.871072893s) [noise=true , address=145.79.227.37:4565 , piece_id=UUNLR2KIA2OB7VK5ETJNMVWMQ6QTOKRUCIIB3XXR4XNNW4DK6KRQ, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:59.197668378 (569h14m59.068751413s) [noise=true , address=45.91.116.13:4154 , piece_id=NFHUV4S3WTCATYYXELPZFRKLGIG7CXGYIOVPHVWLFDWTDYNGSVTQ, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:59.255971015 (569h14m59.12705405s) [noise=true , address=77.90.31.10:4517 , piece_id=T6IGCT265KAHVZOKK3SOROS6DWMLI6ZZVL35LBXGDUCPRBZFQ3YQ, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:59.264492953 (569h14m59.135575998s) [noise=true , address=77.90.31.15:4697 , piece_id=GAELOORMH7VZWU3N6JW4DT2JHQQZQHI2FCF5OHSICND45OAKNMTA, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:59.335803369 (569h14m59.206886414s) [noise=true , address=45.91.116.10:4081 , piece_id=URTDVRL77WJTQ3GFYVWNSNTRTGKG7HGEBH7UCU7MJFZVOXQUYX6A, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:59.660444395 (569h14m59.53152743s) [noise=true , address=45.91.116.9:4261 , piece_id=QIPE6ZI7EZ3PEC7STAVCXGQZ5VTGSB7BUYT6GYBYQBGEBD7XGVYQ, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:59.660464484 (569h14m59.531547519s) [seg_pos={0 0}, upload=499074]: Segment reached optimal threshold of 13 pieces. Dec 29 08:43:59.660789703 (569h14m59.531872738s) [noise=true , address=145.79.227.30:4762 , piece_id=2OX6ZRC6JZACGGDXIZTIYHWY47W5OHUYNIR5QRIIPEGCKDFTDKIA, seg_pos={0 0}, upload=499074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 08:43:59.660797437 (569h14m59.531880472s) [noise=true , address=145.79.227.30:4762 , piece_id=2OX6ZRC6JZACGGDXIZTIYHWY47W5OHUYNIR5QRIIPEGCKDFTDKIA, seg_pos={0 0}, upload=499074]: Long tail cancellation due to optimal threshold Dec 29 08:43:59.660959627 (569h14m59.532042662s) [noise=true , address=45.91.116.12:4226 , piece_id=YP7OX6N4HI5DBNSWKTO3Y5WBEZJ2OZW6NDJC6I4TT267XUAXBVEQ, seg_pos={0 0}, upload=499074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 29 08:43:59.660965287 (569h14m59.532048322s) [noise=true , address=45.91.116.12:4226 , piece_id=YP7OX6N4HI5DBNSWKTO3Y5WBEZJ2OZW6NDJC6I4TT267XUAXBVEQ, seg_pos={0 0}, upload=499074]: Long tail cancellation due to optimal threshold Dec 29 08:43:59.661124540 (569h14m59.532207575s) [noise=true , address=45.91.116.14:4191 , piece_id=VLKKHXI2N6TUR3WST6QIXQTA6QWK5NPFMWTSO73YHNRI3VWJ25XA, seg_pos={0 0}, upload=499074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 08:43:59.661130312 (569h14m59.532213357s) [noise=true , address=45.91.116.14:4191 , piece_id=VLKKHXI2N6TUR3WST6QIXQTA6QWK5NPFMWTSO73YHNRI3VWJ25XA, seg_pos={0 0}, upload=499074]: Long tail cancellation due to optimal threshold Dec 29 08:43:59.661144869 (569h14m59.532227904s) [seg_pos={0 0}, upload=499074]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 08:43:59.661214181 (569h14m59.532297226s) [upload=499074]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:43:59.661219040 (569h14m59.532302075s) [upload=499074]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:43:59.950933373 (569h14m59.822016408s) [upload=499074]: Done uploading segments... Dec 29 08:43:59.950952089 (569h14m59.822035124s) [upload=499074]: Upload finished. err: <nil> Duration: 5.686196443s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X..................................X.......] duration: 27.367647848s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 08:45:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 27.367663749s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..X..X......X..........X.X..X..X.......X............X.......] duration: 5.000807544s trace: https://jaeger.dev.storj.io/trace/2f5d9cf3d6b691bb FAIL (2025-12-29 08:45:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000835516s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X..................................X......] duration: 36.369568242s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 08:44:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 36.369582769s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...XX....X.X..X...X..X...X...X...X.....XXX......X....X..XXX.] duration: 6.010045572s trace: https://jaeger.dev.storj.io/trace/6e8570c9cb7a8654 FAIL (2025-12-29 08:44:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: QIPE6ZI7EZ3PEC7STAVCXGQZ5VTGSB7BUYT6GYBYQBGEBD7XGVYQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: URTDVRL77WJTQ3GFYVWNSNTRTGKG7HGEBH7UCU7MJFZVOXQUYX6A) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: GAELOORMH7VZWU3N6JW4DT2JHQQZQHI2FCF5OHSICND45OAKNMTA) context deadline exceeded ======LOGS====== Dec 29 08:43:56.699530751 (569h14m56.570613786s) [upload=499074]: Starting upload Dec 29 08:43:56.699603459 (569h14m56.570686494s) [upload=499074]: Done starting upload Dec 29 08:43:56.699706125 (569h14m56.57078916s) [upload=499074]: Uploading segments... Dec 29 08:43:56.699824461 (569h14m56.570907496s) [upload=499074]: Got next segment. Inline: false Dec 29 08:43:56.699922538 (569h14m56.571005573s) [upload=499074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 08:43:56.699931024 (569h14m56.571014059s) [upload=499074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 08:43:56.973375414 (569h14m56.844458459s) [seg_pos={0 0}, upload=499074]: Begin upload segment... Dec 29 08:43:56.973432723 (569h14m56.844515758s) [seg_pos={0 0}, upload=499074]: Done begin upload segment. Dec 29 08:43:56.973474783 (569h14m56.844557828s) [noise=true , address=145.79.227.48:4301 , piece_id=EHRCCKPBAS3ZMRDERMTRI536RXN4EAHZ5Y2V67GAJUHUWHWDQX3Q, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.973493448 (569h14m56.844576483s) [noise=true , address=145.79.227.4:4185 , piece_id=M3TVHGCW4K2CN5CYM7XAPBOKOZJ77AON4HKZ4F726Y5YLSEIXWBQ, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.973595763 (569h14m56.844678808s) [noise=true , address=145.79.227.12:4765 , piece_id=VQXSWQUAJQPBKBGJFOL4UHWSRF7L6VBKGO65PDHDIAMQJVATC4RQ, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.974421145 (569h14m56.84550419s) [noise=true , address=145.79.227.11:4729 , piece_id=SFOJPHTKYY3UAAIZLG3EHSYEJPUY7CXBS7YAIANXDK3LAFWEIALA, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.974712920 (569h14m56.845795955s) [noise=true , address=45.91.116.9:4261 , piece_id=QIPE6ZI7EZ3PEC7STAVCXGQZ5VTGSB7BUYT6GYBYQBGEBD7XGVYQ, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.974957005 (569h14m56.84604005s) [noise=true , address=145.79.227.30:4762 , piece_id=2OX6ZRC6JZACGGDXIZTIYHWY47W5OHUYNIR5QRIIPEGCKDFTDKIA, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.975230316 (569h14m56.846313351s) [noise=true , address=45.91.116.12:4226 , piece_id=YP7OX6N4HI5DBNSWKTO3Y5WBEZJ2OZW6NDJC6I4TT267XUAXBVEQ, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.975669773 (569h14m56.846752818s) [noise=true , address=45.91.116.14:4191 , piece_id=VLKKHXI2N6TUR3WST6QIXQTA6QWK5NPFMWTSO73YHNRI3VWJ25XA, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.975776065 (569h14m56.84685911s) [noise=true , address=77.90.31.10:4517 , piece_id=T6IGCT265KAHVZOKK3SOROS6DWMLI6ZZVL35LBXGDUCPRBZFQ3YQ, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.976343445 (569h14m56.84742649s) [noise=true , address=145.79.227.37:4565 , piece_id=UUNLR2KIA2OB7VK5ETJNMVWMQ6QTOKRUCIIB3XXR4XNNW4DK6KRQ, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.976406325 (569h14m56.84748936s) [noise=true , address=45.91.116.10:4081 , piece_id=URTDVRL77WJTQ3GFYVWNSNTRTGKG7HGEBH7UCU7MJFZVOXQUYX6A, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.976830953 (569h14m56.847913988s) [noise=true , address=45.91.116.13:4154 , piece_id=NFHUV4S3WTCATYYXELPZFRKLGIG7CXGYIOVPHVWLFDWTDYNGSVTQ, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.977407381 (569h14m56.848490417s) [noise=true , address=145.79.227.13:4803 , piece_id=PIWP4ZFYI5JK2KOYEYFX6CY6VVYWKUAD2JRMUPU7KMGSGN4HNREQ, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.977933213 (569h14m56.849016248s) [noise=true , address=145.79.227.45:4231 , piece_id=E6NESNDU7TIZ66ZVAUSYMKYHSZAWOCN6PHUJDIFRYV4ROUOI44QA, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.978581717 (569h14m56.849664752s) [noise=true , address=145.79.227.14:4838 , piece_id=VUCUTQUPDYUQOPCCSZJ52KSYZKITWQ5VV6W4KUDEH6OJXNLR34XA, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.979101036 (569h14m56.850184081s) [noise=true , address=77.90.31.15:4697 , piece_id=GAELOORMH7VZWU3N6JW4DT2JHQQZQHI2FCF5OHSICND45OAKNMTA, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:58.668089757 (569h14m58.539172792s) [upload=499074]: Next returned nil segment Dec 29 08:43:58.668107570 (569h14m58.539190615s) [upload=499074]: Waiting for error group managing segments... Dec 29 08:43:58.958856674 (569h14m58.82993972s) [noise=true , address=145.79.227.14:4838 , piece_id=VUCUTQUPDYUQOPCCSZJ52KSYZKITWQ5VV6W4KUDEH6OJXNLR34XA, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:58.960420802 (569h14m58.831503837s) [noise=true , address=145.79.227.4:4185 , piece_id=M3TVHGCW4K2CN5CYM7XAPBOKOZJ77AON4HKZ4F726Y5YLSEIXWBQ, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:58.967931053 (569h14m58.839014098s) [noise=true , address=145.79.227.11:4729 , piece_id=SFOJPHTKYY3UAAIZLG3EHSYEJPUY7CXBS7YAIANXDK3LAFWEIALA, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:58.970437576 (569h14m58.841520611s) [noise=true , address=145.79.227.13:4803 , piece_id=PIWP4ZFYI5JK2KOYEYFX6CY6VVYWKUAD2JRMUPU7KMGSGN4HNREQ, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:58.980239059 (569h14m58.851322104s) [noise=true , address=145.79.227.12:4765 , piece_id=VQXSWQUAJQPBKBGJFOL4UHWSRF7L6VBKGO65PDHDIAMQJVATC4RQ, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:58.980629754 (569h14m58.851712789s) [noise=true , address=145.79.227.45:4231 , piece_id=E6NESNDU7TIZ66ZVAUSYMKYHSZAWOCN6PHUJDIFRYV4ROUOI44QA, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:58.990476994 (569h14m58.861560039s) [noise=true , address=145.79.227.48:4301 , piece_id=EHRCCKPBAS3ZMRDERMTRI536RXN4EAHZ5Y2V67GAJUHUWHWDQX3Q, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:58.999989858 (569h14m58.871072893s) [noise=true , address=145.79.227.37:4565 , piece_id=UUNLR2KIA2OB7VK5ETJNMVWMQ6QTOKRUCIIB3XXR4XNNW4DK6KRQ, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:59.197668378 (569h14m59.068751413s) [noise=true , address=45.91.116.13:4154 , piece_id=NFHUV4S3WTCATYYXELPZFRKLGIG7CXGYIOVPHVWLFDWTDYNGSVTQ, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:59.255971015 (569h14m59.12705405s) [noise=true , address=77.90.31.10:4517 , piece_id=T6IGCT265KAHVZOKK3SOROS6DWMLI6ZZVL35LBXGDUCPRBZFQ3YQ, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:59.264492953 (569h14m59.135575998s) [noise=true , address=77.90.31.15:4697 , piece_id=GAELOORMH7VZWU3N6JW4DT2JHQQZQHI2FCF5OHSICND45OAKNMTA, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:59.335803369 (569h14m59.206886414s) [noise=true , address=45.91.116.10:4081 , piece_id=URTDVRL77WJTQ3GFYVWNSNTRTGKG7HGEBH7UCU7MJFZVOXQUYX6A, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:59.660444395 (569h14m59.53152743s) [noise=true , address=45.91.116.9:4261 , piece_id=QIPE6ZI7EZ3PEC7STAVCXGQZ5VTGSB7BUYT6GYBYQBGEBD7XGVYQ, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:59.660464484 (569h14m59.531547519s) [seg_pos={0 0}, upload=499074]: Segment reached optimal threshold of 13 pieces. Dec 29 08:43:59.660789703 (569h14m59.531872738s) [noise=true , address=145.79.227.30:4762 , piece_id=2OX6ZRC6JZACGGDXIZTIYHWY47W5OHUYNIR5QRIIPEGCKDFTDKIA, seg_pos={0 0}, upload=499074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 08:43:59.660797437 (569h14m59.531880472s) [noise=true , address=145.79.227.30:4762 , piece_id=2OX6ZRC6JZACGGDXIZTIYHWY47W5OHUYNIR5QRIIPEGCKDFTDKIA, seg_pos={0 0}, upload=499074]: Long tail cancellation due to optimal threshold Dec 29 08:43:59.660959627 (569h14m59.532042662s) [noise=true , address=45.91.116.12:4226 , piece_id=YP7OX6N4HI5DBNSWKTO3Y5WBEZJ2OZW6NDJC6I4TT267XUAXBVEQ, seg_pos={0 0}, upload=499074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 29 08:43:59.660965287 (569h14m59.532048322s) [noise=true , address=45.91.116.12:4226 , piece_id=YP7OX6N4HI5DBNSWKTO3Y5WBEZJ2OZW6NDJC6I4TT267XUAXBVEQ, seg_pos={0 0}, upload=499074]: Long tail cancellation due to optimal threshold Dec 29 08:43:59.661124540 (569h14m59.532207575s) [noise=true , address=45.91.116.14:4191 , piece_id=VLKKHXI2N6TUR3WST6QIXQTA6QWK5NPFMWTSO73YHNRI3VWJ25XA, seg_pos={0 0}, upload=499074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 08:43:59.661130312 (569h14m59.532213357s) [noise=true , address=45.91.116.14:4191 , piece_id=VLKKHXI2N6TUR3WST6QIXQTA6QWK5NPFMWTSO73YHNRI3VWJ25XA, seg_pos={0 0}, upload=499074]: Long tail cancellation due to optimal threshold Dec 29 08:43:59.661144869 (569h14m59.532227904s) [seg_pos={0 0}, upload=499074]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 08:43:59.661214181 (569h14m59.532297226s) [upload=499074]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:43:59.661219040 (569h14m59.532302075s) [upload=499074]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:43:59.950933373 (569h14m59.822016408s) [upload=499074]: Done uploading segments... Dec 29 08:43:59.950952089 (569h14m59.822035124s) [upload=499074]: Upload finished. err: <nil> Duration: 6.010072663s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....XX....X.X..X...X..X...X...X...X.....XXX......X....X..XXX] duration: 30.002897218s trace: https://jaeger.dev.storj.io/trace/6e8570c9cb7a8654 FAIL (2025-12-29 08:44:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: QIPE6ZI7EZ3PEC7STAVCXGQZ5VTGSB7BUYT6GYBYQBGEBD7XGVYQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: URTDVRL77WJTQ3GFYVWNSNTRTGKG7HGEBH7UCU7MJFZVOXQUYX6A) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: GAELOORMH7VZWU3N6JW4DT2JHQQZQHI2FCF5OHSICND45OAKNMTA) context deadline exceeded ======LOGS====== Dec 29 08:43:56.699530751 (569h14m56.570613786s) [upload=499074]: Starting upload Dec 29 08:43:56.699603459 (569h14m56.570686494s) [upload=499074]: Done starting upload Dec 29 08:43:56.699706125 (569h14m56.57078916s) [upload=499074]: Uploading segments... Dec 29 08:43:56.699824461 (569h14m56.570907496s) [upload=499074]: Got next segment. Inline: false Dec 29 08:43:56.699922538 (569h14m56.571005573s) [upload=499074]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 08:43:56.699931024 (569h14m56.571014059s) [upload=499074]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 08:43:56.973375414 (569h14m56.844458459s) [seg_pos={0 0}, upload=499074]: Begin upload segment... Dec 29 08:43:56.973432723 (569h14m56.844515758s) [seg_pos={0 0}, upload=499074]: Done begin upload segment. Dec 29 08:43:56.973474783 (569h14m56.844557828s) [noise=true , address=145.79.227.48:4301 , piece_id=EHRCCKPBAS3ZMRDERMTRI536RXN4EAHZ5Y2V67GAJUHUWHWDQX3Q, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.973493448 (569h14m56.844576483s) [noise=true , address=145.79.227.4:4185 , piece_id=M3TVHGCW4K2CN5CYM7XAPBOKOZJ77AON4HKZ4F726Y5YLSEIXWBQ, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.973595763 (569h14m56.844678808s) [noise=true , address=145.79.227.12:4765 , piece_id=VQXSWQUAJQPBKBGJFOL4UHWSRF7L6VBKGO65PDHDIAMQJVATC4RQ, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.974421145 (569h14m56.84550419s) [noise=true , address=145.79.227.11:4729 , piece_id=SFOJPHTKYY3UAAIZLG3EHSYEJPUY7CXBS7YAIANXDK3LAFWEIALA, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.974712920 (569h14m56.845795955s) [noise=true , address=45.91.116.9:4261 , piece_id=QIPE6ZI7EZ3PEC7STAVCXGQZ5VTGSB7BUYT6GYBYQBGEBD7XGVYQ, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.974957005 (569h14m56.84604005s) [noise=true , address=145.79.227.30:4762 , piece_id=2OX6ZRC6JZACGGDXIZTIYHWY47W5OHUYNIR5QRIIPEGCKDFTDKIA, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.975230316 (569h14m56.846313351s) [noise=true , address=45.91.116.12:4226 , piece_id=YP7OX6N4HI5DBNSWKTO3Y5WBEZJ2OZW6NDJC6I4TT267XUAXBVEQ, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.975669773 (569h14m56.846752818s) [noise=true , address=45.91.116.14:4191 , piece_id=VLKKHXI2N6TUR3WST6QIXQTA6QWK5NPFMWTSO73YHNRI3VWJ25XA, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.975776065 (569h14m56.84685911s) [noise=true , address=77.90.31.10:4517 , piece_id=T6IGCT265KAHVZOKK3SOROS6DWMLI6ZZVL35LBXGDUCPRBZFQ3YQ, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.976343445 (569h14m56.84742649s) [noise=true , address=145.79.227.37:4565 , piece_id=UUNLR2KIA2OB7VK5ETJNMVWMQ6QTOKRUCIIB3XXR4XNNW4DK6KRQ, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.976406325 (569h14m56.84748936s) [noise=true , address=45.91.116.10:4081 , piece_id=URTDVRL77WJTQ3GFYVWNSNTRTGKG7HGEBH7UCU7MJFZVOXQUYX6A, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.976830953 (569h14m56.847913988s) [noise=true , address=45.91.116.13:4154 , piece_id=NFHUV4S3WTCATYYXELPZFRKLGIG7CXGYIOVPHVWLFDWTDYNGSVTQ, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.977407381 (569h14m56.848490417s) [noise=true , address=145.79.227.13:4803 , piece_id=PIWP4ZFYI5JK2KOYEYFX6CY6VVYWKUAD2JRMUPU7KMGSGN4HNREQ, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.977933213 (569h14m56.849016248s) [noise=true , address=145.79.227.45:4231 , piece_id=E6NESNDU7TIZ66ZVAUSYMKYHSZAWOCN6PHUJDIFRYV4ROUOI44QA, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.978581717 (569h14m56.849664752s) [noise=true , address=145.79.227.14:4838 , piece_id=VUCUTQUPDYUQOPCCSZJ52KSYZKITWQ5VV6W4KUDEH6OJXNLR34XA, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:56.979101036 (569h14m56.850184081s) [noise=true , address=77.90.31.15:4697 , piece_id=GAELOORMH7VZWU3N6JW4DT2JHQQZQHI2FCF5OHSICND45OAKNMTA, seg_pos={0 0}, upload=499074]: Uploading piece... Dec 29 08:43:58.668089757 (569h14m58.539172792s) [upload=499074]: Next returned nil segment Dec 29 08:43:58.668107570 (569h14m58.539190615s) [upload=499074]: Waiting for error group managing segments... Dec 29 08:43:58.958856674 (569h14m58.82993972s) [noise=true , address=145.79.227.14:4838 , piece_id=VUCUTQUPDYUQOPCCSZJ52KSYZKITWQ5VV6W4KUDEH6OJXNLR34XA, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:58.960420802 (569h14m58.831503837s) [noise=true , address=145.79.227.4:4185 , piece_id=M3TVHGCW4K2CN5CYM7XAPBOKOZJ77AON4HKZ4F726Y5YLSEIXWBQ, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:58.967931053 (569h14m58.839014098s) [noise=true , address=145.79.227.11:4729 , piece_id=SFOJPHTKYY3UAAIZLG3EHSYEJPUY7CXBS7YAIANXDK3LAFWEIALA, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:58.970437576 (569h14m58.841520611s) [noise=true , address=145.79.227.13:4803 , piece_id=PIWP4ZFYI5JK2KOYEYFX6CY6VVYWKUAD2JRMUPU7KMGSGN4HNREQ, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:58.980239059 (569h14m58.851322104s) [noise=true , address=145.79.227.12:4765 , piece_id=VQXSWQUAJQPBKBGJFOL4UHWSRF7L6VBKGO65PDHDIAMQJVATC4RQ, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:58.980629754 (569h14m58.851712789s) [noise=true , address=145.79.227.45:4231 , piece_id=E6NESNDU7TIZ66ZVAUSYMKYHSZAWOCN6PHUJDIFRYV4ROUOI44QA, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:58.990476994 (569h14m58.861560039s) [noise=true , address=145.79.227.48:4301 , piece_id=EHRCCKPBAS3ZMRDERMTRI536RXN4EAHZ5Y2V67GAJUHUWHWDQX3Q, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:58.999989858 (569h14m58.871072893s) [noise=true , address=145.79.227.37:4565 , piece_id=UUNLR2KIA2OB7VK5ETJNMVWMQ6QTOKRUCIIB3XXR4XNNW4DK6KRQ, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:59.197668378 (569h14m59.068751413s) [noise=true , address=45.91.116.13:4154 , piece_id=NFHUV4S3WTCATYYXELPZFRKLGIG7CXGYIOVPHVWLFDWTDYNGSVTQ, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:59.255971015 (569h14m59.12705405s) [noise=true , address=77.90.31.10:4517 , piece_id=T6IGCT265KAHVZOKK3SOROS6DWMLI6ZZVL35LBXGDUCPRBZFQ3YQ, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:59.264492953 (569h14m59.135575998s) [noise=true , address=77.90.31.15:4697 , piece_id=GAELOORMH7VZWU3N6JW4DT2JHQQZQHI2FCF5OHSICND45OAKNMTA, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:59.335803369 (569h14m59.206886414s) [noise=true , address=45.91.116.10:4081 , piece_id=URTDVRL77WJTQ3GFYVWNSNTRTGKG7HGEBH7UCU7MJFZVOXQUYX6A, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:59.660444395 (569h14m59.53152743s) [noise=true , address=45.91.116.9:4261 , piece_id=QIPE6ZI7EZ3PEC7STAVCXGQZ5VTGSB7BUYT6GYBYQBGEBD7XGVYQ, seg_pos={0 0}, upload=499074]: Done uploading piece. err: <nil> Dec 29 08:43:59.660464484 (569h14m59.531547519s) [seg_pos={0 0}, upload=499074]: Segment reached optimal threshold of 13 pieces. Dec 29 08:43:59.660789703 (569h14m59.531872738s) [noise=true , address=145.79.227.30:4762 , piece_id=2OX6ZRC6JZACGGDXIZTIYHWY47W5OHUYNIR5QRIIPEGCKDFTDKIA, seg_pos={0 0}, upload=499074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 08:43:59.660797437 (569h14m59.531880472s) [noise=true , address=145.79.227.30:4762 , piece_id=2OX6ZRC6JZACGGDXIZTIYHWY47W5OHUYNIR5QRIIPEGCKDFTDKIA, seg_pos={0 0}, upload=499074]: Long tail cancellation due to optimal threshold Dec 29 08:43:59.660959627 (569h14m59.532042662s) [noise=true , address=45.91.116.12:4226 , piece_id=YP7OX6N4HI5DBNSWKTO3Y5WBEZJ2OZW6NDJC6I4TT267XUAXBVEQ, seg_pos={0 0}, upload=499074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 29 08:43:59.660965287 (569h14m59.532048322s) [noise=true , address=45.91.116.12:4226 , piece_id=YP7OX6N4HI5DBNSWKTO3Y5WBEZJ2OZW6NDJC6I4TT267XUAXBVEQ, seg_pos={0 0}, upload=499074]: Long tail cancellation due to optimal threshold Dec 29 08:43:59.661124540 (569h14m59.532207575s) [noise=true , address=45.91.116.14:4191 , piece_id=VLKKHXI2N6TUR3WST6QIXQTA6QWK5NPFMWTSO73YHNRI3VWJ25XA, seg_pos={0 0}, upload=499074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 08:43:59.661130312 (569h14m59.532213357s) [noise=true , address=45.91.116.14:4191 , piece_id=VLKKHXI2N6TUR3WST6QIXQTA6QWK5NPFMWTSO73YHNRI3VWJ25XA, seg_pos={0 0}, upload=499074]: Long tail cancellation due to optimal threshold Dec 29 08:43:59.661144869 (569h14m59.532227904s) [seg_pos={0 0}, upload=499074]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 08:43:59.661214181 (569h14m59.532297226s) [upload=499074]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:43:59.661219040 (569h14m59.532302075s) [upload=499074]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:43:59.950933373 (569h14m59.822016408s) [upload=499074]: Done uploading segments... Dec 29 08:43:59.950952089 (569h14m59.822035124s) [upload=499074]: Upload finished. err: <nil> Duration: 30.002908128s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...X..X......X..........X.X..X..X.......X............X......] duration: 4.667897192s trace: https://jaeger.dev.storj.io/trace/2f5d9cf3d6b691bb FAIL (2025-12-29 08:44:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.667926308s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X..................................X.....] duration: 25.707214038s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 08:44:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 25.70723085s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.....XX....X.X..X...X..X...X...X...X.....XXX......X....X..XX] duration: 30.003029879s trace: https://jaeger.dev.storj.io/trace/b1e9d77bf066357 FAIL (2025-12-29 08:43:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 08:43:26.413272541 (569h14m26.284355586s) [upload=499066]: Starting upload Dec 29 08:43:26.413367893 (569h14m26.284450928s) [upload=499066]: Done starting upload Dec 29 08:43:26.413423109 (569h14m26.284506154s) [upload=499066]: Uploading segments... Dec 29 08:43:26.413614703 (569h14m26.284697748s) [upload=499066]: Got next segment. Inline: false Dec 29 08:43:26.413675429 (569h14m26.284758474s) [upload=499066]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 08:43:26.413682753 (569h14m26.284765788s) [upload=499066]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 08:43:26.652179919 (569h14m26.523262974s) [seg_pos={0 0}, upload=499066]: Begin upload segment... Dec 29 08:43:26.652310077 (569h14m26.523393112s) [seg_pos={0 0}, upload=499066]: Done begin upload segment. Dec 29 08:43:26.652388055 (569h14m26.523471101s) [noise=true , address=77.90.31.10:4519 , piece_id=2MTGDESSJXKNHOJLWEFCGGTQA5FSAJ3JGHHCYPJVMZ6H54Y3MPOA, seg_pos={0 0}, upload=499066]: Uploading piece... Dec 29 08:43:26.652423132 (569h14m26.523506177s) [noise=true , address=145.79.227.11:4729 , piece_id=5ZVJCHOB77HXVYP76HYL5OE6D4JQ6XKBS7TPS4XMOUVRNXJMW4ZA, seg_pos={0 0}, upload=499066]: Uploading piece... Dec 29 08:43:26.653208988 (569h14m26.524292033s) [noise=true , address=145.79.227.3:4149 , piece_id=SZ4YIPUCUT4POOCPO3Q4C5VUP6R3IP4CPOMUD35EN6CPGXRWMQFQ, seg_pos={0 0}, upload=499066]: Uploading piece... Dec 29 08:43:26.653377439 (569h14m26.524460484s) [noise=true , address=145.79.227.4:4187 , piece_id=CTLZXIXIRE4LU4U54WZYSIIPF4COLSRWVDHLHZU7ZDO7WVFHGLZQ, seg_pos={0 0}, upload=499066]: Uploading piece... Dec 29 08:43:26.653913891 (569h14m26.524996926s) [noise=true , address=145.79.227.15:4875 , piece_id=4K3T5CT2Y7ZGWF332IQLYXZSNJRAPFDXIGNGZAOL34X2I5LSUGLA, seg_pos={0 0}, upload=499066]: Uploading piece... Dec 29 08:43:26.654350652 (569h14m26.525433697s) [noise=true , address=145.79.227.47:4266 , piece_id=S27PJCNPTJKCOGKOVL2PG4J5ORHZGBFLLWUWFN7NP6C6DWDWENYQ, seg_pos={0 0}, upload=499066]: Uploading piece... Dec 29 08:43:26.654397732 (569h14m26.525480778s) [noise=true , address=45.91.116.10:4082 , piece_id=RHRMPG64HPJULLTXYZFF7TGW4N2OK24EMCF365HYNDN53SRP67MQ, seg_pos={0 0}, upload=499066]: Uploading piece... Dec 29 08:43:26.654804857 (569h14m26.525887892s) [noise=true , address=145.79.227.19:4079 , piece_id=A57FDS6WDYHYPW3L3TP6D4PTPOALUVGZ5CVMACS2YXJGKGKYWXRQ, seg_pos={0 0}, upload=499066]: Uploading piece... Dec 29 08:43:26.655011841 (569h14m26.526094886s) [noise=true , address=77.90.31.19:4842 , piece_id=6YXUGYAZBIZ3VQZQ7JKDBD3OI2YJBK2TJPS6UMSCVWOCY3PTPG2Q, seg_pos={0 0}, upload=499066]: Uploading piece... Dec 29 08:43:26.655205611 (569h14m26.526288656s) [noise=true , address=77.90.31.14:4664 , piece_id=22DSVGDNCTOAZF6ULT4XCPJZDXKDCEHNEIJY7MLYYXS6VH4I2XWA, seg_pos={0 0}, upload=499066]: Uploading piece... Dec 29 08:43:26.655717476 (569h14m26.526800521s) [noise=true , address=45.91.116.13:4154 , piece_id=KWDNSAJWXHOYTNPML4UNSKGQJL5FBA4P6YC7FYKJD6ZRCMOUYU5A, seg_pos={0 0}, upload=499066]: Uploading piece... Dec 29 08:43:26.655750387 (569h14m26.526833432s) [noise=true , address=145.79.227.18:4006 , piece_id=IYR2V7MZASGVHZTTAXCT3EHFZPFM6PP7AG2PSSBLVTWD5C6VQLBQ, seg_pos={0 0}, upload=499066]: Uploading piece... Dec 29 08:43:26.656133497 (569h14m26.527216542s) [noise=true , address=145.79.227.16:4912 , piece_id=FU5343XZFYWCQYFY7RCEDDGG25VSDU3FLIBV7KJ32PBEXF4EBLYQ, seg_pos={0 0}, upload=499066]: Uploading piece... Dec 29 08:43:26.657765815 (569h14m26.52884886s) [noise=true , address=45.91.116.11:4117 , piece_id=DK5VPHGY6QZIVS6JH5YQRMLHDFCOAJCU63PWXKAPLGDFTUHPRUWQ, seg_pos={0 0}, upload=499066]: Uploading piece... Dec 29 08:43:26.658238556 (569h14m26.529321601s) [noise=true , address=145.79.227.25:4618 , piece_id=XJOQ5DBFWRTWUIYDAWZZV2GXUTRG7VGRCM6XXDT5VH24EACTNSUQ, seg_pos={0 0}, upload=499066]: Uploading piece... Dec 29 08:43:26.659276362 (569h14m26.530359407s) [noise=true , address=145.79.227.33:4870 , piece_id=R4SCAIQDPMJHJLJCJFZCBBLT2WJKGHAYTVYCF36GDGNJN5VBC6HA, seg_pos={0 0}, upload=499066]: Uploading piece... Dec 29 08:43:28.348675313 (569h14m28.219758358s) [upload=499066]: Next returned nil segment Dec 29 08:43:28.348693067 (569h14m28.219776102s) [upload=499066]: Waiting for error group managing segments... Dec 29 08:43:28.626832699 (569h14m28.497915734s) [noise=true , address=145.79.227.11:4729 , piece_id=5ZVJCHOB77HXVYP76HYL5OE6D4JQ6XKBS7TPS4XMOUVRNXJMW4ZA, seg_pos={0 0}, upload=499066]: Done uploading piece. err: <nil> Dec 29 08:43:28.632233212 (569h14m28.503316247s) [noise=true , address=145.79.227.25:4618 , piece_id=XJOQ5DBFWRTWUIYDAWZZV2GXUTRG7VGRCM6XXDT5VH24EACTNSUQ, seg_pos={0 0}, upload=499066]: Done uploading piece. err: <nil> Dec 29 08:43:28.643888275 (569h14m28.51497132s) [noise=true , address=145.79.227.18:4006 , piece_id=IYR2V7MZASGVHZTTAXCT3EHFZPFM6PP7AG2PSSBLVTWD5C6VQLBQ, seg_pos={0 0}, upload=499066]: Done uploading piece. err: <nil> Dec 29 08:43:28.644026298 (569h14m28.515109343s) [noise=true , address=145.79.227.16:4912 , piece_id=FU5343XZFYWCQYFY7RCEDDGG25VSDU3FLIBV7KJ32PBEXF4EBLYQ, seg_pos={0 0}, upload=499066]: Done uploading piece. err: <nil> Dec 29 08:43:28.656256025 (569h14m28.52733907s) [noise=true , address=145.79.227.47:4266 , piece_id=S27PJCNPTJKCOGKOVL2PG4J5ORHZGBFLLWUWFN7NP6C6DWDWENYQ, seg_pos={0 0}, upload=499066]: Done uploading piece. err: <nil> Dec 29 08:43:28.658808055 (569h14m28.52989109s) [noise=true , address=145.79.227.3:4149 , piece_id=SZ4YIPUCUT4POOCPO3Q4C5VUP6R3IP4CPOMUD35EN6CPGXRWMQFQ, seg_pos={0 0}, upload=499066]: Done uploading piece. err: <nil> Dec 29 08:43:28.661036308 (569h14m28.532119343s) [noise=true , address=145.79.227.19:4079 , piece_id=A57FDS6WDYHYPW3L3TP6D4PTPOALUVGZ5CVMACS2YXJGKGKYWXRQ, seg_pos={0 0}, upload=499066]: Done uploading piece. err: <nil> Dec 29 08:43:28.670726981 (569h14m28.541810016s) [noise=true , address=145.79.227.15:4875 , piece_id=4K3T5CT2Y7ZGWF332IQLYXZSNJRAPFDXIGNGZAOL34X2I5LSUGLA, seg_pos={0 0}, upload=499066]: Done uploading piece. err: <nil> Dec 29 08:43:28.918421450 (569h14m28.789504496s) [noise=true , address=77.90.31.19:4842 , piece_id=6YXUGYAZBIZ3VQZQ7JKDBD3OI2YJBK2TJPS6UMSCVWOCY3PTPG2Q, seg_pos={0 0}, upload=499066]: Done uploading piece. err: <nil> Dec 29 08:43:28.919631113 (569h14m28.790714158s) [noise=true , address=45.91.116.10:4082 , piece_id=RHRMPG64HPJULLTXYZFF7TGW4N2OK24EMCF365HYNDN53SRP67MQ, seg_pos={0 0}, upload=499066]: Done uploading piece. err: <nil> Dec 29 08:43:28.931581398 (569h14m28.802664443s) [noise=true , address=77.90.31.14:4664 , piece_id=22DSVGDNCTOAZF6ULT4XCPJZDXKDCEHNEIJY7MLYYXS6VH4I2XWA, seg_pos={0 0}, upload=499066]: Done uploading piece. err: <nil> Dec 29 08:43:29.047270158 (569h14m28.918353223s) [noise=true , address=45.91.116.13:4154 , piece_id=KWDNSAJWXHOYTNPML4UNSKGQJL5FBA4P6YC7FYKJD6ZRCMOUYU5A, seg_pos={0 0}, upload=499066]: Done uploading piece. err: <nil> Dec 29 08:43:29.489010492 (569h14m29.360093527s) [noise=true , address=77.90.31.10:4519 , piece_id=2MTGDESSJXKNHOJLWEFCGGTQA5FSAJ3JGHHCYPJVMZ6H54Y3MPOA, seg_pos={0 0}, upload=499066]: Done uploading piece. err: <nil> Dec 29 08:43:29.489032063 (569h14m29.360115098s) [seg_pos={0 0}, upload=499066]: Segment reached optimal threshold of 13 pieces. Dec 29 08:43:29.489265177 (569h14m29.360348232s) [noise=true , address=145.79.227.33:4870 , piece_id=R4SCAIQDPMJHJLJCJFZCBBLT2WJKGHAYTVYCF36GDGNJN5VBC6HA, seg_pos={0 0}, upload=499066]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 29 08:43:29.489273893 (569h14m29.360356948s) [noise=true , address=145.79.227.33:4870 , piece_id=R4SCAIQDPMJHJLJCJFZCBBLT2WJKGHAYTVYCF36GDGNJN5VBC6HA, seg_pos={0 0}, upload=499066]: Long tail cancellation due to optimal threshold Dec 29 08:43:29.489330361 (569h14m29.360413406s) [noise=true , address=45.91.116.11:4117 , piece_id=DK5VPHGY6QZIVS6JH5YQRMLHDFCOAJCU63PWXKAPLGDFTUHPRUWQ, seg_pos={0 0}, upload=499066]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 29 08:43:29.489343766 (569h14m29.360426831s) [noise=true , address=45.91.116.11:4117 , piece_id=DK5VPHGY6QZIVS6JH5YQRMLHDFCOAJCU63PWXKAPLGDFTUHPRUWQ, seg_pos={0 0}, upload=499066]: Long tail cancellation due to optimal threshold Dec 29 08:43:29.489561291 (569h14m29.360644336s) [noise=true , address=145.79.227.4:4187 , piece_id=CTLZXIXIRE4LU4U54WZYSIIPF4COLSRWVDHLHZU7ZDO7WVFHGLZQ, seg_pos={0 0}, upload=499066]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 29 08:43:29.489570258 (569h14m29.360653303s) [noise=true , address=145.79.227.4:4187 , piece_id=CTLZXIXIRE4LU4U54WZYSIIPF4COLSRWVDHLHZU7ZDO7WVFHGLZQ, seg_pos={0 0}, upload=499066]: Long tail cancellation due to optimal threshold Dec 29 08:43:29.489590707 (569h14m29.360673762s) [seg_pos={0 0}, upload=499066]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 08:43:29.489673605 (569h14m29.36075665s) [upload=499066]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:43:29.489679746 (569h14m29.360762792s) [upload=499066]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:43:29.770842475 (569h14m29.64192552s) [upload=499066]: Done uploading segments... Dec 29 08:43:29.770865648 (569h14m29.641948694s) [upload=499066]: Upload finished. err: <nil> Duration: 30.003053656s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X..................................X....] duration: 36.738929609s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 08:43:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 36.738948083s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [....X..X......X..........X.X..X..X.......X............X.....] duration: 3.65533301s trace: https://jaeger.dev.storj.io/trace/2f5d9cf3d6b691bb FAIL (2025-12-29 08:43:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.655355383s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X..................................X...] duration: 15.446942338s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 08:43:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 15.44697429s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [......XX....X.X..X...X..X...X...X...X.....XXX......X....X..X] duration: 30.005569807s trace: https://jaeger.dev.storj.io/trace/4df335da76502be3 FAIL (2025-12-29 08:43:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 7N2L7HBLLL4GJCF2XWXBGAC3GHBURVCESOE2R5SJW7RFBOJE4R5A) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 6IMWBPOUKKSGAWKW632DQNFMHNDS6SVH54S37XR5FLNU4JYA4KPQ) context deadline exceeded ======LOGS====== Dec 29 08:42:38.831982981 (569h13m38.703066026s) [upload=499056]: Starting upload Dec 29 08:42:38.832060299 (569h13m38.703143344s) [upload=499056]: Done starting upload Dec 29 08:42:38.832281350 (569h13m38.703364395s) [upload=499056]: Uploading segments... Dec 29 08:42:38.832542637 (569h13m38.703625672s) [upload=499056]: Got next segment. Inline: false Dec 29 08:42:38.832581351 (569h13m38.703664386s) [upload=499056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 08:42:38.832586521 (569h13m38.703669566s) [upload=499056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 08:42:39.080653679 (569h13m38.951736724s) [seg_pos={0 0}, upload=499056]: Begin upload segment... Dec 29 08:42:39.080715908 (569h13m38.951798943s) [seg_pos={0 0}, upload=499056]: Done begin upload segment. Dec 29 08:42:39.080746737 (569h13m38.951829822s) [noise=true , address=45.91.116.12:4227 , piece_id=R5NSR7ORSGPEIFNTKJPHNEK7RZQXRZRVFMX75YAVFKC4WESCSSVA, seg_pos={0 0}, upload=499056]: Uploading piece... Dec 29 08:42:39.080757878 (569h13m38.951840913s) [noise=true , address=145.79.227.47:4266 , piece_id=YESSNQOV5P7QOYT2ROUFTXKCP7CTCRTPMNYNLJBE6OSWJIMZHV2Q, seg_pos={0 0}, upload=499056]: Uploading piece... Dec 29 08:42:39.080813334 (569h13m38.951896369s) [noise=true , address=45.91.116.6:4976 , piece_id=6IMWBPOUKKSGAWKW632DQNFMHNDS6SVH54S37XR5FLNU4JYA4KPQ, seg_pos={0 0}, upload=499056]: Uploading piece... Dec 29 08:42:39.081009407 (569h13m38.952092472s) [noise=true , address=145.79.227.6:4259 , piece_id=OKRDC3N57UTNLH2QZ3LYTVDPQ6SIX4YEBFURF2NASLQRSX7JIVKA, seg_pos={0 0}, upload=499056]: Uploading piece... Dec 29 08:42:39.081564544 (569h13m38.952647609s) [noise=true , address=145.79.227.3:4151 , piece_id=JIEGJBDZARLMK5XWNGERYAL5O43CKESGPALH5T26RFNCVZP3V5EA, seg_pos={0 0}, upload=499056]: Uploading piece... Dec 29 08:42:39.082114040 (569h13m38.953197075s) [noise=true , address=45.91.116.7:4011 , piece_id=UV2UFTV7R7FCGLZMV3IT64SXRM4VKO2RWJKDKFPYQODCN2ZJWFXA, seg_pos={0 0}, upload=499056]: Uploading piece... Dec 29 08:42:39.083634104 (569h13m38.954717149s) [noise=true , address=145.79.227.2:4114 , piece_id=LQBYTJG7G365BN2RNQK75OST524K3WWMNMI5EKSJJQBOFUPJGP6Q, seg_pos={0 0}, upload=499056]: Uploading piece... Dec 29 08:42:39.084090614 (569h13m38.955173659s) [noise=true , address=45.91.116.13:4155 , piece_id=CAS53XYRGV3CIBZC6SXPF3XEASQXVWLZH35VIZYSWWMF7XTCP7LQ, seg_pos={0 0}, upload=499056]: Uploading piece... Dec 29 08:42:39.084566089 (569h13m38.955649124s) [noise=true , address=145.79.227.39:4013 , piece_id=YZNRNFUTBEU4QOE26WGKYHZ2PPOMBNNTPTAS4IIL3OFZKADFOYYQ, seg_pos={0 0}, upload=499056]: Uploading piece... Dec 29 08:42:39.086497967 (569h13m38.957581012s) [noise=true , address=45.91.116.11:4117 , piece_id=7N2L7HBLLL4GJCF2XWXBGAC3GHBURVCESOE2R5SJW7RFBOJE4R5A, seg_pos={0 0}, upload=499056]: Uploading piece... Dec 29 08:42:39.087950984 (569h13m38.959034029s) [noise=true , address=145.79.227.12:4765 , piece_id=S23NFPIDW46EYQ4KVD3DQLI5OZPL7R57WC74LYERTZTQYOIC4VYQ, seg_pos={0 0}, upload=499056]: Uploading piece... Dec 29 08:42:39.089493472 (569h13m38.960576527s) [noise=true , address=145.79.227.52:4447 , piece_id=PUWCBP6OBGDP5OUCFWJD47WBI7346YAEAXEWGKHJC72H2BY2JQGA, seg_pos={0 0}, upload=499056]: Uploading piece... Dec 29 08:42:39.090990803 (569h13m38.962073848s) [noise=true , address=145.79.227.13:4803 , piece_id=CBN5GP55N7ILETGVWJTBBESYX2HAX4I5T35N2DJF5WFR2WMRG5EQ, seg_pos={0 0}, upload=499056]: Uploading piece... Dec 29 08:42:39.092483164 (569h13m38.963566209s) [noise=true , address=145.79.227.43:4159 , piece_id=UQH7NQZKI6KZBIDSQEY2V7ASRIDV75M6XPQYVKO2PQ466YSPAMAQ, seg_pos={0 0}, upload=499056]: Uploading piece... Dec 29 08:42:39.094037504 (569h13m38.965120539s) [noise=true , address=145.79.227.50:4373 , piece_id=IPFD3UVI75TVBDSVKV4VJ4TZ4BE2YCCY4SEKV726AB6XOFWFD7CA, seg_pos={0 0}, upload=499056]: Uploading piece... Dec 29 08:42:39.094317457 (569h13m38.965400502s) [noise=true , address=77.90.31.19:4841 , piece_id=JP5LWJLTDFGKQ2OHWH6CKTK3NYVMJRHJDSSFBUQMOKELDKEJDSPA, seg_pos={0 0}, upload=499056]: Uploading piece... Dec 29 08:42:40.782315571 (569h13m40.653398606s) [upload=499056]: Next returned nil segment Dec 29 08:42:40.782353143 (569h13m40.653436178s) [upload=499056]: Waiting for error group managing segments... Dec 29 08:42:41.074926909 (569h13m40.946009944s) [noise=true , address=145.79.227.52:4447 , piece_id=PUWCBP6OBGDP5OUCFWJD47WBI7346YAEAXEWGKHJC72H2BY2JQGA, seg_pos={0 0}, upload=499056]: Done uploading piece. err: <nil> Dec 29 08:42:41.077271314 (569h13m40.948354349s) [noise=true , address=145.79.227.3:4151 , piece_id=JIEGJBDZARLMK5XWNGERYAL5O43CKESGPALH5T26RFNCVZP3V5EA, seg_pos={0 0}, upload=499056]: Done uploading piece. err: <nil> Dec 29 08:42:41.078525271 (569h13m40.949608307s) [noise=true , address=145.79.227.2:4114 , piece_id=LQBYTJG7G365BN2RNQK75OST524K3WWMNMI5EKSJJQBOFUPJGP6Q, seg_pos={0 0}, upload=499056]: Done uploading piece. err: <nil> Dec 29 08:42:41.079156073 (569h13m40.950239108s) [noise=true , address=145.79.227.39:4013 , piece_id=YZNRNFUTBEU4QOE26WGKYHZ2PPOMBNNTPTAS4IIL3OFZKADFOYYQ, seg_pos={0 0}, upload=499056]: Done uploading piece. err: <nil> Dec 29 08:42:41.082475313 (569h13m40.953558348s) [noise=true , address=145.79.227.13:4803 , piece_id=CBN5GP55N7ILETGVWJTBBESYX2HAX4I5T35N2DJF5WFR2WMRG5EQ, seg_pos={0 0}, upload=499056]: Done uploading piece. err: <nil> Dec 29 08:42:41.084239242 (569h13m40.955322277s) [noise=true , address=145.79.227.12:4765 , piece_id=S23NFPIDW46EYQ4KVD3DQLI5OZPL7R57WC74LYERTZTQYOIC4VYQ, seg_pos={0 0}, upload=499056]: Done uploading piece. err: <nil> Dec 29 08:42:41.093310905 (569h13m40.96439396s) [noise=true , address=145.79.227.6:4259 , piece_id=OKRDC3N57UTNLH2QZ3LYTVDPQ6SIX4YEBFURF2NASLQRSX7JIVKA, seg_pos={0 0}, upload=499056]: Done uploading piece. err: <nil> Dec 29 08:42:41.100729972 (569h13m40.971813017s) [noise=true , address=145.79.227.47:4266 , piece_id=YESSNQOV5P7QOYT2ROUFTXKCP7CTCRTPMNYNLJBE6OSWJIMZHV2Q, seg_pos={0 0}, upload=499056]: Done uploading piece. err: <nil> Dec 29 08:42:41.102324779 (569h13m40.973407814s) [noise=true , address=145.79.227.50:4373 , piece_id=IPFD3UVI75TVBDSVKV4VJ4TZ4BE2YCCY4SEKV726AB6XOFWFD7CA, seg_pos={0 0}, upload=499056]: Done uploading piece. err: <nil> Dec 29 08:42:41.340114970 (569h13m41.211198005s) [noise=true , address=45.91.116.12:4227 , piece_id=R5NSR7ORSGPEIFNTKJPHNEK7RZQXRZRVFMX75YAVFKC4WESCSSVA, seg_pos={0 0}, upload=499056]: Done uploading piece. err: <nil> Dec 29 08:42:41.357011222 (569h13m41.228094257s) [noise=true , address=45.91.116.11:4117 , piece_id=7N2L7HBLLL4GJCF2XWXBGAC3GHBURVCESOE2R5SJW7RFBOJE4R5A, seg_pos={0 0}, upload=499056]: Done uploading piece. err: <nil> Dec 29 08:42:41.404822021 (569h13m41.275905066s) [noise=true , address=45.91.116.6:4976 , piece_id=6IMWBPOUKKSGAWKW632DQNFMHNDS6SVH54S37XR5FLNU4JYA4KPQ, seg_pos={0 0}, upload=499056]: Done uploading piece. err: <nil> Dec 29 08:42:41.449880049 (569h13m41.320963094s) [noise=true , address=45.91.116.13:4155 , piece_id=CAS53XYRGV3CIBZC6SXPF3XEASQXVWLZH35VIZYSWWMF7XTCP7LQ, seg_pos={0 0}, upload=499056]: Done uploading piece. err: <nil> Dec 29 08:42:41.449901100 (569h13m41.320984145s) [seg_pos={0 0}, upload=499056]: Segment reached optimal threshold of 13 pieces. Dec 29 08:42:41.450072356 (569h13m41.321155391s) [noise=true , address=145.79.227.43:4159 , piece_id=UQH7NQZKI6KZBIDSQEY2V7ASRIDV75M6XPQYVKO2PQ466YSPAMAQ, seg_pos={0 0}, upload=499056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 08:42:41.450080060 (569h13m41.321163095s) [noise=true , address=145.79.227.43:4159 , piece_id=UQH7NQZKI6KZBIDSQEY2V7ASRIDV75M6XPQYVKO2PQ466YSPAMAQ, seg_pos={0 0}, upload=499056]: Long tail cancellation due to optimal threshold Dec 29 08:42:41.450152158 (569h13m41.321235203s) [noise=true , address=77.90.31.19:4841 , piece_id=JP5LWJLTDFGKQ2OHWH6CKTK3NYVMJRHJDSSFBUQMOKELDKEJDSPA, seg_pos={0 0}, upload=499056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 29 08:42:41.450165242 (569h13m41.321248277s) [noise=true , address=77.90.31.19:4841 , piece_id=JP5LWJLTDFGKQ2OHWH6CKTK3NYVMJRHJDSSFBUQMOKELDKEJDSPA, seg_pos={0 0}, upload=499056]: Long tail cancellation due to optimal threshold Dec 29 08:42:41.450249623 (569h13m41.321332658s) [noise=true , address=45.91.116.7:4011 , piece_id=UV2UFTV7R7FCGLZMV3IT64SXRM4VKO2RWJKDKFPYQODCN2ZJWFXA, seg_pos={0 0}, upload=499056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 29 08:42:41.450255645 (569h13m41.32133869s) [noise=true , address=45.91.116.7:4011 , piece_id=UV2UFTV7R7FCGLZMV3IT64SXRM4VKO2RWJKDKFPYQODCN2ZJWFXA, seg_pos={0 0}, upload=499056]: Long tail cancellation due to optimal threshold Dec 29 08:42:41.450302143 (569h13m41.321385178s) [seg_pos={0 0}, upload=499056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 08:42:41.450389490 (569h13m41.321472525s) [upload=499056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:42:41.450394199 (569h13m41.321477244s) [upload=499056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:42:41.737500344 (569h13m41.608583379s) [upload=499056]: Done uploading segments... Dec 29 08:42:41.737522996 (569h13m41.608606041s) [upload=499056]: Upload finished. err: <nil> Duration: 30.005595284s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.....X..X......X..........X.X..X..X.......X............X....] duration: 4.607741176s trace: https://jaeger.dev.storj.io/trace/2f5d9cf3d6b691bb FAIL (2025-12-29 08:42:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.607771533s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X..................................X..] duration: 42.702076715s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 08:42:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 42.702106482s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.......XX....X.X..X...X..X...X...X...X.....XXX......X....X..] duration: 5.652011108s trace: https://jaeger.dev.storj.io/trace/5f92dba6729a91fc FAIL (2025-12-29 08:42:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: NEMN7I3LTTMLZRVANGLHHDPVWCZG3J4IXL2ZRE3E4F5O7Y3UIP7A) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: VZMZXQV5MPAXOJ26YGIOOIWEFRYKVPT24Z3K3F4HHNBZ764LBKNQ) context deadline exceeded ======LOGS====== Dec 29 08:39:56.650486910 (569h10m56.521569946s) [upload=499021]: Starting upload Dec 29 08:39:56.650537087 (569h10m56.521620122s) [upload=499021]: Done starting upload Dec 29 08:39:56.650741516 (569h10m56.521824551s) [upload=499021]: Uploading segments... Dec 29 08:39:56.650787964 (569h10m56.521871029s) [upload=499021]: Got next segment. Inline: false Dec 29 08:39:56.650833671 (569h10m56.521916716s) [upload=499021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 08:39:56.650839062 (569h10m56.521922097s) [upload=499021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 08:39:56.900193510 (569h10m56.771276545s) [seg_pos={0 0}, upload=499021]: Begin upload segment... Dec 29 08:39:56.900231171 (569h10m56.771314206s) [seg_pos={0 0}, upload=499021]: Done begin upload segment. Dec 29 08:39:56.900254386 (569h10m56.771337421s) [noise=true , address=77.90.31.13:4627 , piece_id=NEMN7I3LTTMLZRVANGLHHDPVWCZG3J4IXL2ZRE3E4F5O7Y3UIP7A, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.900379995 (569h10m56.77146303s) [noise=true , address=145.79.227.47:4267 , piece_id=VEV3EMIJZV4E4TLEIBQRKSWG4TL5H2RDWLSWTJDJPYQNBTXWRTCQ, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.900808331 (569h10m56.771891366s) [noise=true , address=145.79.227.45:4229 , piece_id=ZZV6H65EQUVEGCLGX2ZXOVU5NSE5AG5AIHXPXIURRKG7MB4B3KHA, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.901196049 (569h10m56.772279084s) [noise=true , address=145.79.227.51:4410 , piece_id=WV2JCTXE4WF7O7O4OT6RMYQ2QY4MBZHZH4NZ3NDEJB3I7FS5FHGA, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.901462467 (569h10m56.772545502s) [noise=true , address=45.91.116.9:4263 , piece_id=ZHFYQ3BBBR3RATVBYCBPLYFBQJJLFS7R2AOSUL5TUAV2K2OFCVEA, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.901708906 (569h10m56.772791941s) [noise=true , address=145.79.227.29:4942 , piece_id=TALCL5626LLHTEN4L6ZKWIRSC75EDWPNBEL436323UFIXKZXY2JQ, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.901983609 (569h10m56.773066644s) [noise=true , address=145.79.227.3:4149 , piece_id=PCJ5TXDYHQMHMWEHRMNI7CVSKMGQS3F3QOPJZFETNXPHQXTMZMKQ, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.902035978 (569h10m56.773119013s) [noise=true , address=145.79.227.12:4767 , piece_id=R2ZYKSDE3J3BO7JV76JWMRFQPRC3PQRPJOYOGL2DNFPJHYFI6I5Q, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.902225560 (569h10m56.773308595s) [noise=true , address=145.79.227.32:4834 , piece_id=NVZQNJVIVTQ4TJM47ASZV6BAO4NRFR3N6XGM3K2VHZHNPRSK7FQA, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.902339667 (569h10m56.773422702s) [noise=true , address=145.79.227.6:4259 , piece_id=F4HNVYMRRNFHE3XQTQ4FBPSREWYKKRFM37EYZTFLYBEO5XOW45WA, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.902484803 (569h10m56.773567838s) [noise=true , address=45.91.116.4:4940 , piece_id=IHZZA3K5FSTLZZNDUIV7UVDLZEWQ3NOYUZFVYV6JBT4XTYBEKK4Q, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.902623557 (569h10m56.773706592s) [noise=true , address=145.79.227.40:4051 , piece_id=I2UKVW7ESKOLUUFA6NQ3A6LDS7GDFRX2QDGBEFW7SIP2DYNPFXKQ, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.902892199 (569h10m56.773975244s) [noise=true , address=145.79.227.5:4222 , piece_id=RPGNLZPBIC54P2JI23PUYT4PVRJRLRDTA7QM7ZVVQ7TLN2H5TSAQ, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.903157804 (569h10m56.774240839s) [noise=true , address=45.91.116.10:4082 , piece_id=VZMZXQV5MPAXOJ26YGIOOIWEFRYKVPT24Z3K3F4HHNBZ764LBKNQ, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.903512580 (569h10m56.774595615s) [noise=true , address=77.90.31.10:4518 , piece_id=ZQS7G6R7XMR2IXX4B3342PP7ZWWTANUDXCS6PNMDZHWT7D5PVF7A, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.903878266 (569h10m56.774961311s) [noise=true , address=145.79.227.52:4447 , piece_id=J2SURRLGXP3PUGCSSLUUZYE2IJZ6KNQ2GVIXZQ6RHCQIGPP6DGWA, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:58.595994740 (569h10m58.467077775s) [upload=499021]: Next returned nil segment Dec 29 08:39:58.596008867 (569h10m58.467091902s) [upload=499021]: Waiting for error group managing segments... Dec 29 08:39:58.883597649 (569h10m58.754680684s) [noise=true , address=145.79.227.32:4834 , piece_id=NVZQNJVIVTQ4TJM47ASZV6BAO4NRFR3N6XGM3K2VHZHNPRSK7FQA, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:58.890573543 (569h10m58.761656608s) [noise=true , address=145.79.227.6:4259 , piece_id=F4HNVYMRRNFHE3XQTQ4FBPSREWYKKRFM37EYZTFLYBEO5XOW45WA, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:58.891089284 (569h10m58.762172319s) [noise=true , address=145.79.227.29:4942 , piece_id=TALCL5626LLHTEN4L6ZKWIRSC75EDWPNBEL436323UFIXKZXY2JQ, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:58.893264436 (569h10m58.764347481s) [noise=true , address=145.79.227.45:4229 , piece_id=ZZV6H65EQUVEGCLGX2ZXOVU5NSE5AG5AIHXPXIURRKG7MB4B3KHA, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:58.896214453 (569h10m58.767297488s) [noise=true , address=145.79.227.47:4267 , piece_id=VEV3EMIJZV4E4TLEIBQRKSWG4TL5H2RDWLSWTJDJPYQNBTXWRTCQ, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:58.902426873 (569h10m58.773509918s) [noise=true , address=145.79.227.52:4447 , piece_id=J2SURRLGXP3PUGCSSLUUZYE2IJZ6KNQ2GVIXZQ6RHCQIGPP6DGWA, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:58.905168464 (569h10m58.776251499s) [noise=true , address=145.79.227.3:4149 , piece_id=PCJ5TXDYHQMHMWEHRMNI7CVSKMGQS3F3QOPJZFETNXPHQXTMZMKQ, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:58.907507087 (569h10m58.778590122s) [noise=true , address=145.79.227.51:4410 , piece_id=WV2JCTXE4WF7O7O4OT6RMYQ2QY4MBZHZH4NZ3NDEJB3I7FS5FHGA, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:58.908188133 (569h10m58.779271168s) [noise=true , address=145.79.227.12:4767 , piece_id=R2ZYKSDE3J3BO7JV76JWMRFQPRC3PQRPJOYOGL2DNFPJHYFI6I5Q, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:58.910936296 (569h10m58.782019331s) [noise=true , address=145.79.227.5:4222 , piece_id=RPGNLZPBIC54P2JI23PUYT4PVRJRLRDTA7QM7ZVVQ7TLN2H5TSAQ, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:59.090904362 (569h10m58.961987407s) [noise=true , address=77.90.31.13:4627 , piece_id=NEMN7I3LTTMLZRVANGLHHDPVWCZG3J4IXL2ZRE3E4F5O7Y3UIP7A, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:59.110990759 (569h10m58.982073794s) [noise=true , address=45.91.116.10:4082 , piece_id=VZMZXQV5MPAXOJ26YGIOOIWEFRYKVPT24Z3K3F4HHNBZ764LBKNQ, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:59.157684652 (569h10m59.028767687s) [noise=true , address=45.91.116.4:4940 , piece_id=IHZZA3K5FSTLZZNDUIV7UVDLZEWQ3NOYUZFVYV6JBT4XTYBEKK4Q, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:59.157704310 (569h10m59.028787345s) [seg_pos={0 0}, upload=499021]: Segment reached optimal threshold of 13 pieces. Dec 29 08:39:59.158035099 (569h10m59.029118135s) [noise=true , address=77.90.31.10:4518 , piece_id=ZQS7G6R7XMR2IXX4B3342PP7ZWWTANUDXCS6PNMDZHWT7D5PVF7A, seg_pos={0 0}, upload=499021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:59.158042503 (569h10m59.029125548s) [noise=true , address=77.90.31.10:4518 , piece_id=ZQS7G6R7XMR2IXX4B3342PP7ZWWTANUDXCS6PNMDZHWT7D5PVF7A, seg_pos={0 0}, upload=499021]: Long tail cancellation due to optimal threshold Dec 29 08:39:59.158146942 (569h10m59.029229977s) [noise=true , address=145.79.227.40:4051 , piece_id=I2UKVW7ESKOLUUFA6NQ3A6LDS7GDFRX2QDGBEFW7SIP2DYNPFXKQ, seg_pos={0 0}, upload=499021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:59.158152422 (569h10m59.029235458s) [noise=true , address=145.79.227.40:4051 , piece_id=I2UKVW7ESKOLUUFA6NQ3A6LDS7GDFRX2QDGBEFW7SIP2DYNPFXKQ, seg_pos={0 0}, upload=499021]: Long tail cancellation due to optimal threshold Dec 29 08:39:59.158357874 (569h10m59.029440919s) [noise=true , address=45.91.116.9:4263 , piece_id=ZHFYQ3BBBR3RATVBYCBPLYFBQJJLFS7R2AOSUL5TUAV2K2OFCVEA, seg_pos={0 0}, upload=499021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:59.158365189 (569h10m59.029448234s) [noise=true , address=45.91.116.9:4263 , piece_id=ZHFYQ3BBBR3RATVBYCBPLYFBQJJLFS7R2AOSUL5TUAV2K2OFCVEA, seg_pos={0 0}, upload=499021]: Long tail cancellation due to optimal threshold Dec 29 08:39:59.158389414 (569h10m59.029472449s) [seg_pos={0 0}, upload=499021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 08:39:59.158459487 (569h10m59.029542532s) [upload=499021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:59.158463605 (569h10m59.02954664s) [upload=499021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:59.457382132 (569h10m59.328465187s) [upload=499021]: Done uploading segments... Dec 29 08:39:59.457404434 (569h10m59.32848748s) [upload=499021]: Upload finished. err: <nil> Duration: 5.652045723s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [......X..X......X..........X.X..X..X.......X............X...] duration: 3.457343225s trace: https://jaeger.dev.storj.io/trace/2f5d9cf3d6b691bb FAIL (2025-12-29 08:41:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.457371931s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X..................................X.] duration: 44.636437283s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 08:41:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 44.636452041s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [........XX....X.X..X...X..X...X...X...X.....XXX......X....X.] duration: 6.086640188s trace: https://jaeger.dev.storj.io/trace/5f92dba6729a91fc FAIL (2025-12-29 08:41:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: NEMN7I3LTTMLZRVANGLHHDPVWCZG3J4IXL2ZRE3E4F5O7Y3UIP7A) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: VZMZXQV5MPAXOJ26YGIOOIWEFRYKVPT24Z3K3F4HHNBZ764LBKNQ) context deadline exceeded ======LOGS====== Dec 29 08:39:56.650486910 (569h10m56.521569946s) [upload=499021]: Starting upload Dec 29 08:39:56.650537087 (569h10m56.521620122s) [upload=499021]: Done starting upload Dec 29 08:39:56.650741516 (569h10m56.521824551s) [upload=499021]: Uploading segments... Dec 29 08:39:56.650787964 (569h10m56.521871029s) [upload=499021]: Got next segment. Inline: false Dec 29 08:39:56.650833671 (569h10m56.521916716s) [upload=499021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 08:39:56.650839062 (569h10m56.521922097s) [upload=499021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 08:39:56.900193510 (569h10m56.771276545s) [seg_pos={0 0}, upload=499021]: Begin upload segment... Dec 29 08:39:56.900231171 (569h10m56.771314206s) [seg_pos={0 0}, upload=499021]: Done begin upload segment. Dec 29 08:39:56.900254386 (569h10m56.771337421s) [noise=true , address=77.90.31.13:4627 , piece_id=NEMN7I3LTTMLZRVANGLHHDPVWCZG3J4IXL2ZRE3E4F5O7Y3UIP7A, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.900379995 (569h10m56.77146303s) [noise=true , address=145.79.227.47:4267 , piece_id=VEV3EMIJZV4E4TLEIBQRKSWG4TL5H2RDWLSWTJDJPYQNBTXWRTCQ, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.900808331 (569h10m56.771891366s) [noise=true , address=145.79.227.45:4229 , piece_id=ZZV6H65EQUVEGCLGX2ZXOVU5NSE5AG5AIHXPXIURRKG7MB4B3KHA, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.901196049 (569h10m56.772279084s) [noise=true , address=145.79.227.51:4410 , piece_id=WV2JCTXE4WF7O7O4OT6RMYQ2QY4MBZHZH4NZ3NDEJB3I7FS5FHGA, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.901462467 (569h10m56.772545502s) [noise=true , address=45.91.116.9:4263 , piece_id=ZHFYQ3BBBR3RATVBYCBPLYFBQJJLFS7R2AOSUL5TUAV2K2OFCVEA, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.901708906 (569h10m56.772791941s) [noise=true , address=145.79.227.29:4942 , piece_id=TALCL5626LLHTEN4L6ZKWIRSC75EDWPNBEL436323UFIXKZXY2JQ, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.901983609 (569h10m56.773066644s) [noise=true , address=145.79.227.3:4149 , piece_id=PCJ5TXDYHQMHMWEHRMNI7CVSKMGQS3F3QOPJZFETNXPHQXTMZMKQ, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.902035978 (569h10m56.773119013s) [noise=true , address=145.79.227.12:4767 , piece_id=R2ZYKSDE3J3BO7JV76JWMRFQPRC3PQRPJOYOGL2DNFPJHYFI6I5Q, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.902225560 (569h10m56.773308595s) [noise=true , address=145.79.227.32:4834 , piece_id=NVZQNJVIVTQ4TJM47ASZV6BAO4NRFR3N6XGM3K2VHZHNPRSK7FQA, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.902339667 (569h10m56.773422702s) [noise=true , address=145.79.227.6:4259 , piece_id=F4HNVYMRRNFHE3XQTQ4FBPSREWYKKRFM37EYZTFLYBEO5XOW45WA, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.902484803 (569h10m56.773567838s) [noise=true , address=45.91.116.4:4940 , piece_id=IHZZA3K5FSTLZZNDUIV7UVDLZEWQ3NOYUZFVYV6JBT4XTYBEKK4Q, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.902623557 (569h10m56.773706592s) [noise=true , address=145.79.227.40:4051 , piece_id=I2UKVW7ESKOLUUFA6NQ3A6LDS7GDFRX2QDGBEFW7SIP2DYNPFXKQ, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.902892199 (569h10m56.773975244s) [noise=true , address=145.79.227.5:4222 , piece_id=RPGNLZPBIC54P2JI23PUYT4PVRJRLRDTA7QM7ZVVQ7TLN2H5TSAQ, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.903157804 (569h10m56.774240839s) [noise=true , address=45.91.116.10:4082 , piece_id=VZMZXQV5MPAXOJ26YGIOOIWEFRYKVPT24Z3K3F4HHNBZ764LBKNQ, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.903512580 (569h10m56.774595615s) [noise=true , address=77.90.31.10:4518 , piece_id=ZQS7G6R7XMR2IXX4B3342PP7ZWWTANUDXCS6PNMDZHWT7D5PVF7A, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.903878266 (569h10m56.774961311s) [noise=true , address=145.79.227.52:4447 , piece_id=J2SURRLGXP3PUGCSSLUUZYE2IJZ6KNQ2GVIXZQ6RHCQIGPP6DGWA, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:58.595994740 (569h10m58.467077775s) [upload=499021]: Next returned nil segment Dec 29 08:39:58.596008867 (569h10m58.467091902s) [upload=499021]: Waiting for error group managing segments... Dec 29 08:39:58.883597649 (569h10m58.754680684s) [noise=true , address=145.79.227.32:4834 , piece_id=NVZQNJVIVTQ4TJM47ASZV6BAO4NRFR3N6XGM3K2VHZHNPRSK7FQA, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:58.890573543 (569h10m58.761656608s) [noise=true , address=145.79.227.6:4259 , piece_id=F4HNVYMRRNFHE3XQTQ4FBPSREWYKKRFM37EYZTFLYBEO5XOW45WA, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:58.891089284 (569h10m58.762172319s) [noise=true , address=145.79.227.29:4942 , piece_id=TALCL5626LLHTEN4L6ZKWIRSC75EDWPNBEL436323UFIXKZXY2JQ, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:58.893264436 (569h10m58.764347481s) [noise=true , address=145.79.227.45:4229 , piece_id=ZZV6H65EQUVEGCLGX2ZXOVU5NSE5AG5AIHXPXIURRKG7MB4B3KHA, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:58.896214453 (569h10m58.767297488s) [noise=true , address=145.79.227.47:4267 , piece_id=VEV3EMIJZV4E4TLEIBQRKSWG4TL5H2RDWLSWTJDJPYQNBTXWRTCQ, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:58.902426873 (569h10m58.773509918s) [noise=true , address=145.79.227.52:4447 , piece_id=J2SURRLGXP3PUGCSSLUUZYE2IJZ6KNQ2GVIXZQ6RHCQIGPP6DGWA, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:58.905168464 (569h10m58.776251499s) [noise=true , address=145.79.227.3:4149 , piece_id=PCJ5TXDYHQMHMWEHRMNI7CVSKMGQS3F3QOPJZFETNXPHQXTMZMKQ, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:58.907507087 (569h10m58.778590122s) [noise=true , address=145.79.227.51:4410 , piece_id=WV2JCTXE4WF7O7O4OT6RMYQ2QY4MBZHZH4NZ3NDEJB3I7FS5FHGA, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:58.908188133 (569h10m58.779271168s) [noise=true , address=145.79.227.12:4767 , piece_id=R2ZYKSDE3J3BO7JV76JWMRFQPRC3PQRPJOYOGL2DNFPJHYFI6I5Q, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:58.910936296 (569h10m58.782019331s) [noise=true , address=145.79.227.5:4222 , piece_id=RPGNLZPBIC54P2JI23PUYT4PVRJRLRDTA7QM7ZVVQ7TLN2H5TSAQ, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:59.090904362 (569h10m58.961987407s) [noise=true , address=77.90.31.13:4627 , piece_id=NEMN7I3LTTMLZRVANGLHHDPVWCZG3J4IXL2ZRE3E4F5O7Y3UIP7A, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:59.110990759 (569h10m58.982073794s) [noise=true , address=45.91.116.10:4082 , piece_id=VZMZXQV5MPAXOJ26YGIOOIWEFRYKVPT24Z3K3F4HHNBZ764LBKNQ, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:59.157684652 (569h10m59.028767687s) [noise=true , address=45.91.116.4:4940 , piece_id=IHZZA3K5FSTLZZNDUIV7UVDLZEWQ3NOYUZFVYV6JBT4XTYBEKK4Q, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:59.157704310 (569h10m59.028787345s) [seg_pos={0 0}, upload=499021]: Segment reached optimal threshold of 13 pieces. Dec 29 08:39:59.158035099 (569h10m59.029118135s) [noise=true , address=77.90.31.10:4518 , piece_id=ZQS7G6R7XMR2IXX4B3342PP7ZWWTANUDXCS6PNMDZHWT7D5PVF7A, seg_pos={0 0}, upload=499021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:59.158042503 (569h10m59.029125548s) [noise=true , address=77.90.31.10:4518 , piece_id=ZQS7G6R7XMR2IXX4B3342PP7ZWWTANUDXCS6PNMDZHWT7D5PVF7A, seg_pos={0 0}, upload=499021]: Long tail cancellation due to optimal threshold Dec 29 08:39:59.158146942 (569h10m59.029229977s) [noise=true , address=145.79.227.40:4051 , piece_id=I2UKVW7ESKOLUUFA6NQ3A6LDS7GDFRX2QDGBEFW7SIP2DYNPFXKQ, seg_pos={0 0}, upload=499021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:59.158152422 (569h10m59.029235458s) [noise=true , address=145.79.227.40:4051 , piece_id=I2UKVW7ESKOLUUFA6NQ3A6LDS7GDFRX2QDGBEFW7SIP2DYNPFXKQ, seg_pos={0 0}, upload=499021]: Long tail cancellation due to optimal threshold Dec 29 08:39:59.158357874 (569h10m59.029440919s) [noise=true , address=45.91.116.9:4263 , piece_id=ZHFYQ3BBBR3RATVBYCBPLYFBQJJLFS7R2AOSUL5TUAV2K2OFCVEA, seg_pos={0 0}, upload=499021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:59.158365189 (569h10m59.029448234s) [noise=true , address=45.91.116.9:4263 , piece_id=ZHFYQ3BBBR3RATVBYCBPLYFBQJJLFS7R2AOSUL5TUAV2K2OFCVEA, seg_pos={0 0}, upload=499021]: Long tail cancellation due to optimal threshold Dec 29 08:39:59.158389414 (569h10m59.029472449s) [seg_pos={0 0}, upload=499021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 08:39:59.158459487 (569h10m59.029542532s) [upload=499021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:59.158463605 (569h10m59.02954664s) [upload=499021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:59.457382132 (569h10m59.328465187s) [upload=499021]: Done uploading segments... Dec 29 08:39:59.457404434 (569h10m59.32848748s) [upload=499021]: Upload finished. err: <nil> Duration: 6.086671387s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.......X..X......X..........X.X..X..X.......X............X..] duration: 3.363000668s trace: https://jaeger.dev.storj.io/trace/2f5d9cf3d6b691bb FAIL (2025-12-29 08:40:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.363047758s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X..................................X] duration: 1m0.002895965s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 08:40:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 1m0.002957972s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.........XX....X.X..X...X..X...X...X...X.....XXX......X....X] duration: 30.003826638s trace: https://jaeger.dev.storj.io/trace/5f92dba6729a91fc FAIL (2025-12-29 08:40:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: NEMN7I3LTTMLZRVANGLHHDPVWCZG3J4IXL2ZRE3E4F5O7Y3UIP7A) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: VZMZXQV5MPAXOJ26YGIOOIWEFRYKVPT24Z3K3F4HHNBZ764LBKNQ) context deadline exceeded ======LOGS====== Dec 29 08:39:56.650486910 (569h10m56.521569946s) [upload=499021]: Starting upload Dec 29 08:39:56.650537087 (569h10m56.521620122s) [upload=499021]: Done starting upload Dec 29 08:39:56.650741516 (569h10m56.521824551s) [upload=499021]: Uploading segments... Dec 29 08:39:56.650787964 (569h10m56.521871029s) [upload=499021]: Got next segment. Inline: false Dec 29 08:39:56.650833671 (569h10m56.521916716s) [upload=499021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 08:39:56.650839062 (569h10m56.521922097s) [upload=499021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 08:39:56.900193510 (569h10m56.771276545s) [seg_pos={0 0}, upload=499021]: Begin upload segment... Dec 29 08:39:56.900231171 (569h10m56.771314206s) [seg_pos={0 0}, upload=499021]: Done begin upload segment. Dec 29 08:39:56.900254386 (569h10m56.771337421s) [noise=true , address=77.90.31.13:4627 , piece_id=NEMN7I3LTTMLZRVANGLHHDPVWCZG3J4IXL2ZRE3E4F5O7Y3UIP7A, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.900379995 (569h10m56.77146303s) [noise=true , address=145.79.227.47:4267 , piece_id=VEV3EMIJZV4E4TLEIBQRKSWG4TL5H2RDWLSWTJDJPYQNBTXWRTCQ, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.900808331 (569h10m56.771891366s) [noise=true , address=145.79.227.45:4229 , piece_id=ZZV6H65EQUVEGCLGX2ZXOVU5NSE5AG5AIHXPXIURRKG7MB4B3KHA, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.901196049 (569h10m56.772279084s) [noise=true , address=145.79.227.51:4410 , piece_id=WV2JCTXE4WF7O7O4OT6RMYQ2QY4MBZHZH4NZ3NDEJB3I7FS5FHGA, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.901462467 (569h10m56.772545502s) [noise=true , address=45.91.116.9:4263 , piece_id=ZHFYQ3BBBR3RATVBYCBPLYFBQJJLFS7R2AOSUL5TUAV2K2OFCVEA, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.901708906 (569h10m56.772791941s) [noise=true , address=145.79.227.29:4942 , piece_id=TALCL5626LLHTEN4L6ZKWIRSC75EDWPNBEL436323UFIXKZXY2JQ, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.901983609 (569h10m56.773066644s) [noise=true , address=145.79.227.3:4149 , piece_id=PCJ5TXDYHQMHMWEHRMNI7CVSKMGQS3F3QOPJZFETNXPHQXTMZMKQ, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.902035978 (569h10m56.773119013s) [noise=true , address=145.79.227.12:4767 , piece_id=R2ZYKSDE3J3BO7JV76JWMRFQPRC3PQRPJOYOGL2DNFPJHYFI6I5Q, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.902225560 (569h10m56.773308595s) [noise=true , address=145.79.227.32:4834 , piece_id=NVZQNJVIVTQ4TJM47ASZV6BAO4NRFR3N6XGM3K2VHZHNPRSK7FQA, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.902339667 (569h10m56.773422702s) [noise=true , address=145.79.227.6:4259 , piece_id=F4HNVYMRRNFHE3XQTQ4FBPSREWYKKRFM37EYZTFLYBEO5XOW45WA, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.902484803 (569h10m56.773567838s) [noise=true , address=45.91.116.4:4940 , piece_id=IHZZA3K5FSTLZZNDUIV7UVDLZEWQ3NOYUZFVYV6JBT4XTYBEKK4Q, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.902623557 (569h10m56.773706592s) [noise=true , address=145.79.227.40:4051 , piece_id=I2UKVW7ESKOLUUFA6NQ3A6LDS7GDFRX2QDGBEFW7SIP2DYNPFXKQ, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.902892199 (569h10m56.773975244s) [noise=true , address=145.79.227.5:4222 , piece_id=RPGNLZPBIC54P2JI23PUYT4PVRJRLRDTA7QM7ZVVQ7TLN2H5TSAQ, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.903157804 (569h10m56.774240839s) [noise=true , address=45.91.116.10:4082 , piece_id=VZMZXQV5MPAXOJ26YGIOOIWEFRYKVPT24Z3K3F4HHNBZ764LBKNQ, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.903512580 (569h10m56.774595615s) [noise=true , address=77.90.31.10:4518 , piece_id=ZQS7G6R7XMR2IXX4B3342PP7ZWWTANUDXCS6PNMDZHWT7D5PVF7A, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:56.903878266 (569h10m56.774961311s) [noise=true , address=145.79.227.52:4447 , piece_id=J2SURRLGXP3PUGCSSLUUZYE2IJZ6KNQ2GVIXZQ6RHCQIGPP6DGWA, seg_pos={0 0}, upload=499021]: Uploading piece... Dec 29 08:39:58.595994740 (569h10m58.467077775s) [upload=499021]: Next returned nil segment Dec 29 08:39:58.596008867 (569h10m58.467091902s) [upload=499021]: Waiting for error group managing segments... Dec 29 08:39:58.883597649 (569h10m58.754680684s) [noise=true , address=145.79.227.32:4834 , piece_id=NVZQNJVIVTQ4TJM47ASZV6BAO4NRFR3N6XGM3K2VHZHNPRSK7FQA, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:58.890573543 (569h10m58.761656608s) [noise=true , address=145.79.227.6:4259 , piece_id=F4HNVYMRRNFHE3XQTQ4FBPSREWYKKRFM37EYZTFLYBEO5XOW45WA, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:58.891089284 (569h10m58.762172319s) [noise=true , address=145.79.227.29:4942 , piece_id=TALCL5626LLHTEN4L6ZKWIRSC75EDWPNBEL436323UFIXKZXY2JQ, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:58.893264436 (569h10m58.764347481s) [noise=true , address=145.79.227.45:4229 , piece_id=ZZV6H65EQUVEGCLGX2ZXOVU5NSE5AG5AIHXPXIURRKG7MB4B3KHA, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:58.896214453 (569h10m58.767297488s) [noise=true , address=145.79.227.47:4267 , piece_id=VEV3EMIJZV4E4TLEIBQRKSWG4TL5H2RDWLSWTJDJPYQNBTXWRTCQ, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:58.902426873 (569h10m58.773509918s) [noise=true , address=145.79.227.52:4447 , piece_id=J2SURRLGXP3PUGCSSLUUZYE2IJZ6KNQ2GVIXZQ6RHCQIGPP6DGWA, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:58.905168464 (569h10m58.776251499s) [noise=true , address=145.79.227.3:4149 , piece_id=PCJ5TXDYHQMHMWEHRMNI7CVSKMGQS3F3QOPJZFETNXPHQXTMZMKQ, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:58.907507087 (569h10m58.778590122s) [noise=true , address=145.79.227.51:4410 , piece_id=WV2JCTXE4WF7O7O4OT6RMYQ2QY4MBZHZH4NZ3NDEJB3I7FS5FHGA, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:58.908188133 (569h10m58.779271168s) [noise=true , address=145.79.227.12:4767 , piece_id=R2ZYKSDE3J3BO7JV76JWMRFQPRC3PQRPJOYOGL2DNFPJHYFI6I5Q, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:58.910936296 (569h10m58.782019331s) [noise=true , address=145.79.227.5:4222 , piece_id=RPGNLZPBIC54P2JI23PUYT4PVRJRLRDTA7QM7ZVVQ7TLN2H5TSAQ, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:59.090904362 (569h10m58.961987407s) [noise=true , address=77.90.31.13:4627 , piece_id=NEMN7I3LTTMLZRVANGLHHDPVWCZG3J4IXL2ZRE3E4F5O7Y3UIP7A, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:59.110990759 (569h10m58.982073794s) [noise=true , address=45.91.116.10:4082 , piece_id=VZMZXQV5MPAXOJ26YGIOOIWEFRYKVPT24Z3K3F4HHNBZ764LBKNQ, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:59.157684652 (569h10m59.028767687s) [noise=true , address=45.91.116.4:4940 , piece_id=IHZZA3K5FSTLZZNDUIV7UVDLZEWQ3NOYUZFVYV6JBT4XTYBEKK4Q, seg_pos={0 0}, upload=499021]: Done uploading piece. err: <nil> Dec 29 08:39:59.157704310 (569h10m59.028787345s) [seg_pos={0 0}, upload=499021]: Segment reached optimal threshold of 13 pieces. Dec 29 08:39:59.158035099 (569h10m59.029118135s) [noise=true , address=77.90.31.10:4518 , piece_id=ZQS7G6R7XMR2IXX4B3342PP7ZWWTANUDXCS6PNMDZHWT7D5PVF7A, seg_pos={0 0}, upload=499021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:59.158042503 (569h10m59.029125548s) [noise=true , address=77.90.31.10:4518 , piece_id=ZQS7G6R7XMR2IXX4B3342PP7ZWWTANUDXCS6PNMDZHWT7D5PVF7A, seg_pos={0 0}, upload=499021]: Long tail cancellation due to optimal threshold Dec 29 08:39:59.158146942 (569h10m59.029229977s) [noise=true , address=145.79.227.40:4051 , piece_id=I2UKVW7ESKOLUUFA6NQ3A6LDS7GDFRX2QDGBEFW7SIP2DYNPFXKQ, seg_pos={0 0}, upload=499021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:59.158152422 (569h10m59.029235458s) [noise=true , address=145.79.227.40:4051 , piece_id=I2UKVW7ESKOLUUFA6NQ3A6LDS7GDFRX2QDGBEFW7SIP2DYNPFXKQ, seg_pos={0 0}, upload=499021]: Long tail cancellation due to optimal threshold Dec 29 08:39:59.158357874 (569h10m59.029440919s) [noise=true , address=45.91.116.9:4263 , piece_id=ZHFYQ3BBBR3RATVBYCBPLYFBQJJLFS7R2AOSUL5TUAV2K2OFCVEA, seg_pos={0 0}, upload=499021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:59.158365189 (569h10m59.029448234s) [noise=true , address=45.91.116.9:4263 , piece_id=ZHFYQ3BBBR3RATVBYCBPLYFBQJJLFS7R2AOSUL5TUAV2K2OFCVEA, seg_pos={0 0}, upload=499021]: Long tail cancellation due to optimal threshold Dec 29 08:39:59.158389414 (569h10m59.029472449s) [seg_pos={0 0}, upload=499021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 08:39:59.158459487 (569h10m59.029542532s) [upload=499021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:59.158463605 (569h10m59.02954664s) [upload=499021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:59.457382132 (569h10m59.328465187s) [upload=499021]: Done uploading segments... Dec 29 08:39:59.457404434 (569h10m59.32848748s) [upload=499021]: Upload finished. err: <nil> Duration: 30.003846867s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [........X..X......X..........X.X..X..X.......X............X.] duration: 3.591346482s trace: https://jaeger.dev.storj.io/trace/2f5d9cf3d6b691bb FAIL (2025-12-29 08:39:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.591414231s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.........X..X......X..........X.X..X..X.......X............X] duration: 5.001848911s trace: https://jaeger.dev.storj.io/trace/2f5d9cf3d6b691bb FAIL (2025-12-29 08:39:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001872105s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..........X..X......X..........X.X..X..X.......X............] duration: 3.459420608s trace: https://jaeger.dev.storj.io/trace/4ea4b41656f8a3e1 FAIL (2025-12-29 08:39:03 UTC) orbiter warning: context deadline exceeded Duration: 3.459452089s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 29.910802844s trace: https://jaeger.dev.storj.io/trace/a6340b01349e0d FAIL (2025-12-29 08:38:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.13:4645 , piece_id=FFHY4TLTCB6HF4PDYGRIRCJLTHHCZLBUJSGB7L7T4H67GK7J6EMA, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.247819992 (568h45m36.118903027s) [noise=true , address=145.79.227.29:4946 , piece_id=KOGEA4IIZVDMURP4XUX4XTZLP7FFIFVUSCGOG3O6DSML6JTESZOQ, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.248342738 (568h45m36.119425793s) [noise=true , address=77.90.31.14:4665 , piece_id=5TVEIZ6ND2NCI4DZKVTBC5J7WOLHTYZTTCOFH3BPZGUFRHMVL3OQ, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.248847809 (568h45m36.119930944s) [noise=true , address=64.49.13.169:4709 , piece_id=TTYPDIIT662HPX3XANNYFQFR2XV63NE4OS2HZMUYF7BNLG2LFA6A, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.249356217 (568h45m36.120439272s) [noise=true , address=145.79.227.44:4201 , piece_id=ZA2P5IKR7GYGFJYOBWMLE6CRCEUFCTANYCLX7QSUAHEVZEYHWHLA, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.249944006 (568h45m36.121027041s) [noise=true , address=77.90.31.16:4750 , piece_id=GMPTFGSLUU4G65ASVJL5B7S3WRCYJRJWCNBQTC2APVLEIGRD2PMA, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.250484676 (568h45m36.121567731s) [noise=true , address=64.49.13.140:4513 , piece_id=YCCKSXVZVIYUWVLCVMFJP5M5AHXRNKMMRELPEXI6DYSBOIZYDDHA, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.252307867 (568h45m36.123390912s) [noise=true , address=145.79.227.34:4930 , piece_id=KS6U2OTGZ5L3CZ2M6HC4GZZEJYSRZCB75SMKLVVCWQLIBBV7AZYQ, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.254200411 (568h45m36.125283456s) [noise=true , address=145.79.227.33:4884 , piece_id=TF7KDEVRAL54N7ZMQ32B7IB5CY6LDRPNN72DMSWXVO3N2ELU622A, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.256317642 (568h45m36.127400707s) [noise=true , address=145.79.227.6:4275 , piece_id=HCFQBSFOAHWUFX577XBGRGXEM5BA2G4LUZKTNXAGDZS6QE2NC6KA, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.258310357 (568h45m36.129393412s) [noise=true , address=145.79.227.30:4772 , piece_id=F4M7OXPFSOGSOJTVYJ5TVRCNSMK2JA7HLP3XMVCTY5QM52RGIN6Q, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.258396260 (568h45m36.129479306s) [noise=true , address=77.90.31.18:4832 , piece_id=LJBNOAJWM3CBF6URNX3AUBBNWVC3RWFGTVIU5CWS5GI5R6VFECOA, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.259535260 (568h45m36.130618305s) [noise=true , address=45.91.116.6:4986 , piece_id=Z7EAQNBEEAD7PG64B72GNJ55JT2GXTCD3MA2WZDS5QJQVQTGRQQQ, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.260603623 (568h45m36.131686658s) [noise=true , address=45.91.116.14:4207 , piece_id=6UG4LG76D7DLBPXDGMHW7RLXYBJQXJUHJIGE7RWGB3FPUYEHPJNQ, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.261823867 (568h45m36.132906902s) [noise=true , address=145.79.227.17:4988 , piece_id=FTQAV3Z2SKB4OQXK3AR47Z3POPO4WHKQQTUL3RL7M3W6ZYNVN2KA, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.261964755 (568h45m36.1330478s) [noise=true , address=145.79.227.7:4302 , piece_id=672P4QLMXU4J6AYBMG2F5TID3T3PBDZ5KHSFTWVSIUSWKQ2ALQDQ, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:37.701154511 (568h45m37.572237546s) [upload=498639]: Next returned nil segment Dec 29 08:14:37.701167917 (568h45m37.572250962s) [upload=498639]: Waiting for error group managing segments... Dec 29 08:14:38.099583464 (568h45m37.9706665s) [noise=true , address=145.79.227.17:4988 , piece_id=FTQAV3Z2SKB4OQXK3AR47Z3POPO4WHKQQTUL3RL7M3W6ZYNVN2KA, seg_pos={0 0}, upload=498639]: Done uploading piece. err: <nil> Dec 29 08:14:38.107512071 (568h45m37.978595106s) [noise=true , address=145.79.227.19:4091 , piece_id=N7ZJCMJQO7N6WLQ6G333XFU5OGMJL2THXXC6MOLBOFZGXCLQXWQQ, seg_pos={0 0}, upload=498639]: Done uploading piece. err: <nil> Dec 29 08:14:38.118701377 (568h45m37.989784412s) [noise=true , address=64.49.13.140:4513 , piece_id=YCCKSXVZVIYUWVLCVMFJP5M5AHXRNKMMRELPEXI6DYSBOIZYDDHA, seg_pos={0 0}, upload=498639]: Done uploading piece. err: <nil> Dec 29 08:14:38.121990390 (568h45m37.993073436s) [noise=true , address=145.79.227.6:4275 , piece_id=HCFQBSFOAHWUFX577XBGRGXEM5BA2G4LUZKTNXAGDZS6QE2NC6KA, seg_pos={0 0}, upload=498639]: Done uploading piece. err: <nil> Dec 29 08:14:38.130817759 (568h45m38.001900804s) [noise=true , address=145.79.227.29:4946 , piece_id=KOGEA4IIZVDMURP4XUX4XTZLP7FFIFVUSCGOG3O6DSML6JTESZOQ, seg_pos={0 0}, upload=498639]: Done uploading piece. err: <nil> Dec 29 08:14:38.140792001 (568h45m38.011875036s) [noise=true , address=145.79.227.7:4302 , piece_id=672P4QLMXU4J6AYBMG2F5TID3T3PBDZ5KHSFTWVSIUSWKQ2ALQDQ, seg_pos={0 0}, upload=498639]: Done uploading piece. err: <nil> Dec 29 08:14:38.308459122 (568h45m38.179542168s) [noise=true , address=77.90.31.14:4665 , piece_id=5TVEIZ6ND2NCI4DZKVTBC5J7WOLHTYZTTCOFH3BPZGUFRHMVL3OQ, seg_pos={0 0}, upload=498639]: Done uploading piece. err: <nil> Dec 29 08:14:38.352666790 (568h45m38.223749825s) [noise=true , address=77.90.31.18:4832 , piece_id=LJBNOAJWM3CBF6URNX3AUBBNWVC3RWFGTVIU5CWS5GI5R6VFECOA, seg_pos={0 0}, upload=498639]: Done uploading piece. err: <nil> Dec 29 08:14:38.448694279 (568h45m38.319777324s) [noise=true , address=77.90.31.15:4724 , piece_id=6FFXZPBGIFVPSU3B5722L7TD4K6JJERWLGJND5PAMAJAIZQMGSSQ, seg_pos={0 0}, upload=498639]: Done uploading piece. err: <nil> Dec 29 08:14:38.455222761 (568h45m38.326305806s) [noise=true , address=45.91.116.14:4207 , piece_id=6UG4LG76D7DLBPXDGMHW7RLXYBJQXJUHJIGE7RWGB3FPUYEHPJNQ, seg_pos={0 0}, upload=498639]: Done uploading piece. err: <nil> Dec 29 08:14:38.496227930 (568h45m38.367310975s) [noise=true , address=45.91.116.6:4986 , piece_id=Z7EAQNBEEAD7PG64B72GNJ55JT2GXTCD3MA2WZDS5QJQVQTGRQQQ, seg_pos={0 0}, upload=498639]: Done uploading piece. err: <nil> Dec 29 08:14:38.975868381 (568h45m38.846951417s) [noise=true , address=145.79.227.34:4930 , piece_id=KS6U2OTGZ5L3CZ2M6HC4GZZEJYSRZCB75SMKLVVCWQLIBBV7AZYQ, seg_pos={0 0}, upload=498639]: Done uploading piece. err: <nil> Dec 29 08:14:39.027529080 (568h45m38.898612125s) [noise=true , address=145.79.227.30:4772 , piece_id=F4M7OXPFSOGSOJTVYJ5TVRCNSMK2JA7HLP3XMVCTY5QM52RGIN6Q, seg_pos={0 0}, upload=498639]: Done uploading piece. err: <nil> Dec 29 08:14:39.059807343 (568h45m38.930890389s) [noise=true , address=145.79.227.44:4201 , piece_id=ZA2P5IKR7GYGFJYOBWMLE6CRCEUFCTANYCLX7QSUAHEVZEYHWHLA, seg_pos={0 0}, upload=498639]: Done uploading piece. err: <nil> Dec 29 08:14:39.059832932 (568h45m38.930915967s) [seg_pos={0 0}, upload=498639]: Segment reached optimal threshold of 14 pieces. Dec 29 08:14:39.060092186 (568h45m38.931175241s) [noise=true , address=77.90.31.13:4645 , piece_id=FFHY4TLTCB6HF4PDYGRIRCJLTHHCZLBUJSGB7L7T4H67GK7J6EMA, seg_pos={0 0}, upload=498639]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg): protocol: expected piece hash; context canceled; EOF Dec 29 08:14:39.060105542 (568h45m38.931188587s) [noise=true , address=77.90.31.13:4645 , piece_id=FFHY4TLTCB6HF4PDYGRIRCJLTHHCZLBUJSGB7L7T4H67GK7J6EMA, seg_pos={0 0}, upload=498639]: Long tail cancellation due to optimal threshold Dec 29 08:14:39.060153042 (568h45m38.931236077s) [noise=true , address=64.49.13.169:4709 , piece_id=TTYPDIIT662HPX3XANNYFQFR2XV63NE4OS2HZMUYF7BNLG2LFA6A, seg_pos={0 0}, upload=498639]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:40112->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:40112->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40112->64.49.13.169:4709: use of closed network connection Dec 29 08:14:39.060170705 (568h45m38.93125375s) [noise=true , address=64.49.13.169:4709 , piece_id=TTYPDIIT662HPX3XANNYFQFR2XV63NE4OS2HZMUYF7BNLG2LFA6A, seg_pos={0 0}, upload=498639]: Long tail cancellation due to optimal threshold Dec 29 08:14:39.060407406 (568h45m38.931490441s) [noise=true , address=145.79.227.33:4884 , piece_id=TF7KDEVRAL54N7ZMQ32B7IB5CY6LDRPNN72DMSWXVO3N2ELU622A, seg_pos={0 0}, upload=498639]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG): protocol: expected piece hash; context canceled; EOF Dec 29 08:14:39.060413978 (568h45m38.931497023s) [noise=true , address=145.79.227.33:4884 , piece_id=TF7KDEVRAL54N7ZMQ32B7IB5CY6LDRPNN72DMSWXVO3N2ELU622A, seg_pos={0 0}, upload=498639]: Long tail cancellation due to optimal threshold Dec 29 08:14:39.060447702 (568h45m38.931530747s) [noise=true , address=77.90.31.16:4750 , piece_id=GMPTFGSLUU4G65ASVJL5B7S3WRCYJRJWCNBQTC2APVLEIGRD2PMA, seg_pos={0 0}, upload=498639]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE): protocol: expected piece hash; context canceled; EOF Dec 29 08:14:39.060456971 (568h45m38.931540016s) [noise=true , address=77.90.31.16:4750 , piece_id=GMPTFGSLUU4G65ASVJL5B7S3WRCYJRJWCNBQTC2APVLEIGRD2PMA, seg_pos={0 0}, upload=498639]: Long tail cancellation due to optimal threshold Dec 29 08:14:39.060490455 (568h45m38.9315735s) [seg_pos={0 0}, upload=498639]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:14:39.060588581 (568h45m38.931671616s) [upload=498639]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:14:39.060594803 (568h45m38.931677848s) [upload=498639]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:14:39.360082939 (568h45m39.231165974s) [upload=498639]: Done uploading segments... Dec 29 08:14:39.360106114 (568h45m39.231189159s) [upload=498639]: Upload finished. err: <nil> Duration: 29.910820908s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..........XX....X.X..X...X..X...X...X...X.....XXX......X....] duration: 5.910517617s trace: https://jaeger.dev.storj.io/trace/374dcd82fa994bdb FAIL (2025-12-29 08:38:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: 6QYPMT6UEB7ZTUG2EY3CSQY37KDBOVX7NMDOC4SBMVR2NPBEK2BQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: CKKZO7ZTIGAQZ2NHULIGM5VZ76NLPUZJWLMKRWB34BDKRSRWGPIA) context deadline exceeded ======LOGS====== 08:35:25.767301467 (569h6m25.638384512s) [upload=498956]: Starting upload Dec 29 08:35:25.767377582 (569h6m25.638460627s) [upload=498956]: Done starting upload Dec 29 08:35:25.767575269 (569h6m25.638658314s) [upload=498956]: Uploading segments... Dec 29 08:35:25.767658557 (569h6m25.638741612s) [upload=498956]: Got next segment. Inline: false Dec 29 08:35:25.767695708 (569h6m25.638778743s) [upload=498956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 08:35:25.767701839 (569h6m25.638784874s) [upload=498956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 08:35:26.012438005 (569h6m25.88352105s) [seg_pos={0 0}, upload=498956]: Begin upload segment... Dec 29 08:35:26.012492127 (569h6m25.883575162s) [seg_pos={0 0}, upload=498956]: Done begin upload segment. Dec 29 08:35:26.012553204 (569h6m25.883636239s) [noise=true , address=145.79.227.6:4258 , piece_id=M5TXYMDXODXXNZLFUE4TENVUJERH6M6P3N2S6HFHWVLJMT3453AA, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.012574194 (569h6m25.883657239s) [noise=true , address=45.91.116.7:4010 , piece_id=CKKZO7ZTIGAQZ2NHULIGM5VZ76NLPUZJWLMKRWB34BDKRSRWGPIA, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.013040662 (569h6m25.884123707s) [noise=true , address=77.90.31.19:4842 , piece_id=TAG2JA7BG2PEFZNSKMD3LYTU3FIIWILS46BRNZ6T22EIWMSEYZNQ, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.013327508 (569h6m25.884410543s) [noise=true , address=145.79.227.52:4447 , piece_id=SRAT6RFRJBE6WKPJERYMJQJD6W2UIUC5AB4JYZWTRVDHU7LV74PA, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.013611078 (569h6m25.884694113s) [noise=true , address=77.90.31.10:4517 , piece_id=IVC6UZ5E7XJ6JZCIJ725JJYDLZXTDEJS7FHYYETZ4P3L2ZNBQ6QQ, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.013927621 (569h6m25.885010666s) [noise=true , address=45.91.116.9:4264 , piece_id=GTJKAG5VAQXIY3DUSWC37ASCS6GGUC4HV6SHSQCDHOOHXVTIRBCA, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.015007457 (569h6m25.886090503s) [noise=true , address=145.79.227.16:4910 , piece_id=JWG3LVMM2QGC5YHUMBOGG45LERASIIWQVLWGPJMNKRRSYT3M7OYA, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.015304422 (569h6m25.886387457s) [noise=true , address=45.91.116.8:4045 , piece_id=JW5GV6RDF4TSDYCGK335V4U7FWZ35ZSOVR2ZTX4GGS2VGWJU4NCA, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.016535587 (569h6m25.887618632s) [noise=true , address=45.91.116.5:4299 , piece_id=NBHMOKJZH5TN543EZCYWZIFZ3LYKRJ3BSRLOEMG4INJ5VS7K63KQ, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.017629620 (569h6m25.888712655s) [noise=true , address=145.79.227.19:4077 , piece_id=DMXTHNHIMWQYKZCP4CY4MWS4GHAHH5PWVXI7Q73E6C5HLQP6JSSQ, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.018767277 (569h6m25.889850312s) [noise=true , address=145.79.227.48:4302 , piece_id=WPINK2W3URTINL2KXID5JDIMIPJBOAWDPYBNCVVU6DRVB2QMBO4Q, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.019844548 (569h6m25.890927593s) [noise=true , address=45.91.116.13:4156 , piece_id=2AERQMVUL7JIIL6LPSYMZHG7WSNZCORB5IUPCJ5K6K5MVJ6XOBPA, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.019941272 (569h6m25.891024307s) [noise=true , address=45.91.116.14:4190 , piece_id=6GFGFQ256LTLOUXMOERJMIBLPW3WTM3NRWA4BT6JLH5NJ54L2O4Q, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.021027521 (569h6m25.892110566s) [noise=true , address=145.79.227.11:4730 , piece_id=3GFJ5HEXCQR24DVL3QQDOBNPM52XRTJOYQSGC7YEZXIPPQHFNXIQ, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.021216440 (569h6m25.892299475s) [noise=true , address=145.79.227.49:4337 , piece_id=3WGEPNMRHQ4WCZYCE7JLUIJEZZ6HVDGA7EQBK2CSQ6FXIN6Z3BYA, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.022330972 (569h6m25.893414017s) [noise=true , address=45.91.116.11:4120 , piece_id=6QYPMT6UEB7ZTUG2EY3CSQY37KDBOVX7NMDOC4SBMVR2NPBEK2BQ, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:27.705558259 (569h6m27.576641304s) [upload=498956]: Next returned nil segment Dec 29 08:35:27.705576423 (569h6m27.576659458s) [upload=498956]: Waiting for error group managing segments... Dec 29 08:35:27.997360655 (569h6m27.86844369s) [noise=true , address=145.79.227.52:4447 , piece_id=SRAT6RFRJBE6WKPJERYMJQJD6W2UIUC5AB4JYZWTRVDHU7LV74PA, seg_pos={0 0}, upload=498956]: Done uploading piece. err: <nil> Dec 29 08:35:28.004807876 (569h6m27.875890931s) [noise=true , address=145.79.227.16:4910 , piece_id=JWG3LVMM2QGC5YHUMBOGG45LERASIIWQVLWGPJMNKRRSYT3M7OYA, seg_pos={0 0}, upload=498956]: Done uploading piece. err: <nil> Dec 29 08:35:28.025316346 (569h6m27.896399381s) [noise=true , address=145.79.227.48:4302 , piece_id=WPINK2W3URTINL2KXID5JDIMIPJBOAWDPYBNCVVU6DRVB2QMBO4Q, seg_pos={0 0}, upload=498956]: Done uploading piece. err: <nil> Dec 29 08:35:28.026523225 (569h6m27.89760626s) [noise=true , address=145.79.227.11:4730 , piece_id=3GFJ5HEXCQR24DVL3QQDOBNPM52XRTJOYQSGC7YEZXIPPQHFNXIQ, seg_pos={0 0}, upload=498956]: Done uploading piece. err: <nil> Dec 29 08:35:28.038168099 (569h6m27.909251144s) [noise=true , address=145.79.227.19:4077 , piece_id=DMXTHNHIMWQYKZCP4CY4MWS4GHAHH5PWVXI7Q73E6C5HLQP6JSSQ, seg_pos={0 0}, upload=498956]: Done uploading piece. err: <nil> Dec 29 08:35:28.053740540 (569h6m27.924823585s) [noise=true , address=145.79.227.6:4258 , piece_id=M5TXYMDXODXXNZLFUE4TENVUJERH6M6P3N2S6HFHWVLJMT3453AA, seg_pos={0 0}, upload=498956]: Done uploading piece. err: <nil> Dec 29 08:35:28.210002928 (569h6m28.081085963s) [noise=true , address=77.90.31.10:4517 , piece_id=IVC6UZ5E7XJ6JZCIJ725JJYDLZXTDEJS7FHYYETZ4P3L2ZNBQ6QQ, seg_pos={0 0}, upload=498956]: Done uploading piece. err: <nil> Dec 29 08:35:28.264562077 (569h6m28.135645112s) [noise=true , address=45.91.116.9:4264 , piece_id=GTJKAG5VAQXIY3DUSWC37ASCS6GGUC4HV6SHSQCDHOOHXVTIRBCA, seg_pos={0 0}, upload=498956]: Done uploading piece. err: <nil> Dec 29 08:35:28.272373861 (569h6m28.143456896s) [noise=true , address=45.91.116.11:4120 , piece_id=6QYPMT6UEB7ZTUG2EY3CSQY37KDBOVX7NMDOC4SBMVR2NPBEK2BQ, seg_pos={0 0}, upload=498956]: Done uploading piece. err: <nil> Dec 29 08:35:28.356661147 (569h6m28.227744202s) [noise=true , address=45.91.116.7:4010 , piece_id=CKKZO7ZTIGAQZ2NHULIGM5VZ76NLPUZJWLMKRWB34BDKRSRWGPIA, seg_pos={0 0}, upload=498956]: Done uploading piece. err: <nil> Dec 29 08:35:28.365782435 (569h6m28.23686547s) [noise=true , address=45.91.116.5:4299 , piece_id=NBHMOKJZH5TN543EZCYWZIFZ3LYKRJ3BSRLOEMG4INJ5VS7K63KQ, seg_pos={0 0}, upload=498956]: Done uploading piece. err: <nil> Dec 29 08:35:28.444213861 (569h6m28.315296906s) [noise=true , address=45.91.116.13:4156 , piece_id=2AERQMVUL7JIIL6LPSYMZHG7WSNZCORB5IUPCJ5K6K5MVJ6XOBPA, seg_pos={0 0}, upload=498956]: Done uploading piece. err: <nil> Dec 29 08:35:28.457104276 (569h6m28.328187351s) [noise=true , address=45.91.116.14:4190 , piece_id=6GFGFQ256LTLOUXMOERJMIBLPW3WTM3NRWA4BT6JLH5NJ54L2O4Q, seg_pos={0 0}, upload=498956]: Done uploading piece. err: <nil> Dec 29 08:35:28.457133382 (569h6m28.328216427s) [seg_pos={0 0}, upload=498956]: Segment reached optimal threshold of 13 pieces. Dec 29 08:35:28.457344904 (569h6m28.328427969s) [noise=true , address=45.91.116.8:4045 , piece_id=JW5GV6RDF4TSDYCGK335V4U7FWZ35ZSOVR2ZTX4GGS2VGWJU4NCA, seg_pos={0 0}, upload=498956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:36024->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:36024->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36024->45.91.116.8:4045: use of closed network connection Dec 29 08:35:28.457352378 (569h6m28.328435413s) [noise=true , address=45.91.116.8:4045 , piece_id=JW5GV6RDF4TSDYCGK335V4U7FWZ35ZSOVR2ZTX4GGS2VGWJU4NCA, seg_pos={0 0}, upload=498956]: Long tail cancellation due to optimal threshold Dec 29 08:35:28.457528494 (569h6m28.328611539s) [noise=true , address=145.79.227.49:4337 , piece_id=3WGEPNMRHQ4WCZYCE7JLUIJEZZ6HVDGA7EQBK2CSQ6FXIN6Z3BYA, seg_pos={0 0}, upload=498956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 29 08:35:28.457544004 (569h6m28.328627039s) [noise=true , address=145.79.227.49:4337 , piece_id=3WGEPNMRHQ4WCZYCE7JLUIJEZZ6HVDGA7EQBK2CSQ6FXIN6Z3BYA, seg_pos={0 0}, upload=498956]: Long tail cancellation due to optimal threshold Dec 29 08:35:28.457738013 (569h6m28.328821058s) [noise=true , address=77.90.31.19:4842 , piece_id=TAG2JA7BG2PEFZNSKMD3LYTU3FIIWILS46BRNZ6T22EIWMSEYZNQ, seg_pos={0 0}, upload=498956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: read tcp 188.166.225.170:44186->77.90.31.19:4842: use of closed network connection; context canceled Dec 29 08:35:28.457746508 (569h6m28.328829554s) [noise=true , address=77.90.31.19:4842 , piece_id=TAG2JA7BG2PEFZNSKMD3LYTU3FIIWILS46BRNZ6T22EIWMSEYZNQ, seg_pos={0 0}, upload=498956]: Long tail cancellation due to optimal threshold Dec 29 08:35:28.457765465 (569h6m28.3288485s) [seg_pos={0 0}, upload=498956]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 08:35:28.457879251 (569h6m28.328962286s) [upload=498956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:35:28.457884461 (569h6m28.328967496s) [upload=498956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:35:28.751199387 (569h6m28.622282423s) [upload=498956]: Done uploading segments... Dec 29 08:35:28.751221018 (569h6m28.622304063s) [upload=498956]: Upload finished. err: <nil> Duration: 5.910544909s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 35.580614603s trace: https://jaeger.dev.storj.io/trace/a6340b01349e0d FAIL (2025-12-29 08:38:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.13:4645 , piece_id=FFHY4TLTCB6HF4PDYGRIRCJLTHHCZLBUJSGB7L7T4H67GK7J6EMA, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.247819992 (568h45m36.118903027s) [noise=true , address=145.79.227.29:4946 , piece_id=KOGEA4IIZVDMURP4XUX4XTZLP7FFIFVUSCGOG3O6DSML6JTESZOQ, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.248342738 (568h45m36.119425793s) [noise=true , address=77.90.31.14:4665 , piece_id=5TVEIZ6ND2NCI4DZKVTBC5J7WOLHTYZTTCOFH3BPZGUFRHMVL3OQ, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.248847809 (568h45m36.119930944s) [noise=true , address=64.49.13.169:4709 , piece_id=TTYPDIIT662HPX3XANNYFQFR2XV63NE4OS2HZMUYF7BNLG2LFA6A, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.249356217 (568h45m36.120439272s) [noise=true , address=145.79.227.44:4201 , piece_id=ZA2P5IKR7GYGFJYOBWMLE6CRCEUFCTANYCLX7QSUAHEVZEYHWHLA, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.249944006 (568h45m36.121027041s) [noise=true , address=77.90.31.16:4750 , piece_id=GMPTFGSLUU4G65ASVJL5B7S3WRCYJRJWCNBQTC2APVLEIGRD2PMA, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.250484676 (568h45m36.121567731s) [noise=true , address=64.49.13.140:4513 , piece_id=YCCKSXVZVIYUWVLCVMFJP5M5AHXRNKMMRELPEXI6DYSBOIZYDDHA, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.252307867 (568h45m36.123390912s) [noise=true , address=145.79.227.34:4930 , piece_id=KS6U2OTGZ5L3CZ2M6HC4GZZEJYSRZCB75SMKLVVCWQLIBBV7AZYQ, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.254200411 (568h45m36.125283456s) [noise=true , address=145.79.227.33:4884 , piece_id=TF7KDEVRAL54N7ZMQ32B7IB5CY6LDRPNN72DMSWXVO3N2ELU622A, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.256317642 (568h45m36.127400707s) [noise=true , address=145.79.227.6:4275 , piece_id=HCFQBSFOAHWUFX577XBGRGXEM5BA2G4LUZKTNXAGDZS6QE2NC6KA, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.258310357 (568h45m36.129393412s) [noise=true , address=145.79.227.30:4772 , piece_id=F4M7OXPFSOGSOJTVYJ5TVRCNSMK2JA7HLP3XMVCTY5QM52RGIN6Q, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.258396260 (568h45m36.129479306s) [noise=true , address=77.90.31.18:4832 , piece_id=LJBNOAJWM3CBF6URNX3AUBBNWVC3RWFGTVIU5CWS5GI5R6VFECOA, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.259535260 (568h45m36.130618305s) [noise=true , address=45.91.116.6:4986 , piece_id=Z7EAQNBEEAD7PG64B72GNJ55JT2GXTCD3MA2WZDS5QJQVQTGRQQQ, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.260603623 (568h45m36.131686658s) [noise=true , address=45.91.116.14:4207 , piece_id=6UG4LG76D7DLBPXDGMHW7RLXYBJQXJUHJIGE7RWGB3FPUYEHPJNQ, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.261823867 (568h45m36.132906902s) [noise=true , address=145.79.227.17:4988 , piece_id=FTQAV3Z2SKB4OQXK3AR47Z3POPO4WHKQQTUL3RL7M3W6ZYNVN2KA, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:36.261964755 (568h45m36.1330478s) [noise=true , address=145.79.227.7:4302 , piece_id=672P4QLMXU4J6AYBMG2F5TID3T3PBDZ5KHSFTWVSIUSWKQ2ALQDQ, seg_pos={0 0}, upload=498639]: Uploading piece... Dec 29 08:14:37.701154511 (568h45m37.572237546s) [upload=498639]: Next returned nil segment Dec 29 08:14:37.701167917 (568h45m37.572250962s) [upload=498639]: Waiting for error group managing segments... Dec 29 08:14:38.099583464 (568h45m37.9706665s) [noise=true , address=145.79.227.17:4988 , piece_id=FTQAV3Z2SKB4OQXK3AR47Z3POPO4WHKQQTUL3RL7M3W6ZYNVN2KA, seg_pos={0 0}, upload=498639]: Done uploading piece. err: <nil> Dec 29 08:14:38.107512071 (568h45m37.978595106s) [noise=true , address=145.79.227.19:4091 , piece_id=N7ZJCMJQO7N6WLQ6G333XFU5OGMJL2THXXC6MOLBOFZGXCLQXWQQ, seg_pos={0 0}, upload=498639]: Done uploading piece. err: <nil> Dec 29 08:14:38.118701377 (568h45m37.989784412s) [noise=true , address=64.49.13.140:4513 , piece_id=YCCKSXVZVIYUWVLCVMFJP5M5AHXRNKMMRELPEXI6DYSBOIZYDDHA, seg_pos={0 0}, upload=498639]: Done uploading piece. err: <nil> Dec 29 08:14:38.121990390 (568h45m37.993073436s) [noise=true , address=145.79.227.6:4275 , piece_id=HCFQBSFOAHWUFX577XBGRGXEM5BA2G4LUZKTNXAGDZS6QE2NC6KA, seg_pos={0 0}, upload=498639]: Done uploading piece. err: <nil> Dec 29 08:14:38.130817759 (568h45m38.001900804s) [noise=true , address=145.79.227.29:4946 , piece_id=KOGEA4IIZVDMURP4XUX4XTZLP7FFIFVUSCGOG3O6DSML6JTESZOQ, seg_pos={0 0}, upload=498639]: Done uploading piece. err: <nil> Dec 29 08:14:38.140792001 (568h45m38.011875036s) [noise=true , address=145.79.227.7:4302 , piece_id=672P4QLMXU4J6AYBMG2F5TID3T3PBDZ5KHSFTWVSIUSWKQ2ALQDQ, seg_pos={0 0}, upload=498639]: Done uploading piece. err: <nil> Dec 29 08:14:38.308459122 (568h45m38.179542168s) [noise=true , address=77.90.31.14:4665 , piece_id=5TVEIZ6ND2NCI4DZKVTBC5J7WOLHTYZTTCOFH3BPZGUFRHMVL3OQ, seg_pos={0 0}, upload=498639]: Done uploading piece. err: <nil> Dec 29 08:14:38.352666790 (568h45m38.223749825s) [noise=true , address=77.90.31.18:4832 , piece_id=LJBNOAJWM3CBF6URNX3AUBBNWVC3RWFGTVIU5CWS5GI5R6VFECOA, seg_pos={0 0}, upload=498639]: Done uploading piece. err: <nil> Dec 29 08:14:38.448694279 (568h45m38.319777324s) [noise=true , address=77.90.31.15:4724 , piece_id=6FFXZPBGIFVPSU3B5722L7TD4K6JJERWLGJND5PAMAJAIZQMGSSQ, seg_pos={0 0}, upload=498639]: Done uploading piece. err: <nil> Dec 29 08:14:38.455222761 (568h45m38.326305806s) [noise=true , address=45.91.116.14:4207 , piece_id=6UG4LG76D7DLBPXDGMHW7RLXYBJQXJUHJIGE7RWGB3FPUYEHPJNQ, seg_pos={0 0}, upload=498639]: Done uploading piece. err: <nil> Dec 29 08:14:38.496227930 (568h45m38.367310975s) [noise=true , address=45.91.116.6:4986 , piece_id=Z7EAQNBEEAD7PG64B72GNJ55JT2GXTCD3MA2WZDS5QJQVQTGRQQQ, seg_pos={0 0}, upload=498639]: Done uploading piece. err: <nil> Dec 29 08:14:38.975868381 (568h45m38.846951417s) [noise=true , address=145.79.227.34:4930 , piece_id=KS6U2OTGZ5L3CZ2M6HC4GZZEJYSRZCB75SMKLVVCWQLIBBV7AZYQ, seg_pos={0 0}, upload=498639]: Done uploading piece. err: <nil> Dec 29 08:14:39.027529080 (568h45m38.898612125s) [noise=true , address=145.79.227.30:4772 , piece_id=F4M7OXPFSOGSOJTVYJ5TVRCNSMK2JA7HLP3XMVCTY5QM52RGIN6Q, seg_pos={0 0}, upload=498639]: Done uploading piece. err: <nil> Dec 29 08:14:39.059807343 (568h45m38.930890389s) [noise=true , address=145.79.227.44:4201 , piece_id=ZA2P5IKR7GYGFJYOBWMLE6CRCEUFCTANYCLX7QSUAHEVZEYHWHLA, seg_pos={0 0}, upload=498639]: Done uploading piece. err: <nil> Dec 29 08:14:39.059832932 (568h45m38.930915967s) [seg_pos={0 0}, upload=498639]: Segment reached optimal threshold of 14 pieces. Dec 29 08:14:39.060092186 (568h45m38.931175241s) [noise=true , address=77.90.31.13:4645 , piece_id=FFHY4TLTCB6HF4PDYGRIRCJLTHHCZLBUJSGB7L7T4H67GK7J6EMA, seg_pos={0 0}, upload=498639]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg): protocol: expected piece hash; context canceled; EOF Dec 29 08:14:39.060105542 (568h45m38.931188587s) [noise=true , address=77.90.31.13:4645 , piece_id=FFHY4TLTCB6HF4PDYGRIRCJLTHHCZLBUJSGB7L7T4H67GK7J6EMA, seg_pos={0 0}, upload=498639]: Long tail cancellation due to optimal threshold Dec 29 08:14:39.060153042 (568h45m38.931236077s) [noise=true , address=64.49.13.169:4709 , piece_id=TTYPDIIT662HPX3XANNYFQFR2XV63NE4OS2HZMUYF7BNLG2LFA6A, seg_pos={0 0}, upload=498639]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:40112->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:40112->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40112->64.49.13.169:4709: use of closed network connection Dec 29 08:14:39.060170705 (568h45m38.93125375s) [noise=true , address=64.49.13.169:4709 , piece_id=TTYPDIIT662HPX3XANNYFQFR2XV63NE4OS2HZMUYF7BNLG2LFA6A, seg_pos={0 0}, upload=498639]: Long tail cancellation due to optimal threshold Dec 29 08:14:39.060407406 (568h45m38.931490441s) [noise=true , address=145.79.227.33:4884 , piece_id=TF7KDEVRAL54N7ZMQ32B7IB5CY6LDRPNN72DMSWXVO3N2ELU622A, seg_pos={0 0}, upload=498639]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qGXdnm83P4eMnCtZdRt2x5yAWP7pQafWwoPG72daZJVR9pyKG): protocol: expected piece hash; context canceled; EOF Dec 29 08:14:39.060413978 (568h45m38.931497023s) [noise=true , address=145.79.227.33:4884 , piece_id=TF7KDEVRAL54N7ZMQ32B7IB5CY6LDRPNN72DMSWXVO3N2ELU622A, seg_pos={0 0}, upload=498639]: Long tail cancellation due to optimal threshold Dec 29 08:14:39.060447702 (568h45m38.931530747s) [noise=true , address=77.90.31.16:4750 , piece_id=GMPTFGSLUU4G65ASVJL5B7S3WRCYJRJWCNBQTC2APVLEIGRD2PMA, seg_pos={0 0}, upload=498639]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE): protocol: expected piece hash; context canceled; EOF Dec 29 08:14:39.060456971 (568h45m38.931540016s) [noise=true , address=77.90.31.16:4750 , piece_id=GMPTFGSLUU4G65ASVJL5B7S3WRCYJRJWCNBQTC2APVLEIGRD2PMA, seg_pos={0 0}, upload=498639]: Long tail cancellation due to optimal threshold Dec 29 08:14:39.060490455 (568h45m38.9315735s) [seg_pos={0 0}, upload=498639]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:14:39.060588581 (568h45m38.931671616s) [upload=498639]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:14:39.060594803 (568h45m38.931677848s) [upload=498639]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:14:39.360082939 (568h45m39.231165974s) [upload=498639]: Done uploading segments... Dec 29 08:14:39.360106114 (568h45m39.231189159s) [upload=498639]: Upload finished. err: <nil> Duration: 35.580639771s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...........XX....X.X..X...X..X...X...X...X.....XXX......X...] duration: 5.701487338s trace: https://jaeger.dev.storj.io/trace/374dcd82fa994bdb FAIL (2025-12-29 08:38:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: 6QYPMT6UEB7ZTUG2EY3CSQY37KDBOVX7NMDOC4SBMVR2NPBEK2BQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: CKKZO7ZTIGAQZ2NHULIGM5VZ76NLPUZJWLMKRWB34BDKRSRWGPIA) context deadline exceeded ======LOGS====== 08:35:25.767301467 (569h6m25.638384512s) [upload=498956]: Starting upload Dec 29 08:35:25.767377582 (569h6m25.638460627s) [upload=498956]: Done starting upload Dec 29 08:35:25.767575269 (569h6m25.638658314s) [upload=498956]: Uploading segments... Dec 29 08:35:25.767658557 (569h6m25.638741612s) [upload=498956]: Got next segment. Inline: false Dec 29 08:35:25.767695708 (569h6m25.638778743s) [upload=498956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 08:35:25.767701839 (569h6m25.638784874s) [upload=498956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 08:35:26.012438005 (569h6m25.88352105s) [seg_pos={0 0}, upload=498956]: Begin upload segment... Dec 29 08:35:26.012492127 (569h6m25.883575162s) [seg_pos={0 0}, upload=498956]: Done begin upload segment. Dec 29 08:35:26.012553204 (569h6m25.883636239s) [noise=true , address=145.79.227.6:4258 , piece_id=M5TXYMDXODXXNZLFUE4TENVUJERH6M6P3N2S6HFHWVLJMT3453AA, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.012574194 (569h6m25.883657239s) [noise=true , address=45.91.116.7:4010 , piece_id=CKKZO7ZTIGAQZ2NHULIGM5VZ76NLPUZJWLMKRWB34BDKRSRWGPIA, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.013040662 (569h6m25.884123707s) [noise=true , address=77.90.31.19:4842 , piece_id=TAG2JA7BG2PEFZNSKMD3LYTU3FIIWILS46BRNZ6T22EIWMSEYZNQ, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.013327508 (569h6m25.884410543s) [noise=true , address=145.79.227.52:4447 , piece_id=SRAT6RFRJBE6WKPJERYMJQJD6W2UIUC5AB4JYZWTRVDHU7LV74PA, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.013611078 (569h6m25.884694113s) [noise=true , address=77.90.31.10:4517 , piece_id=IVC6UZ5E7XJ6JZCIJ725JJYDLZXTDEJS7FHYYETZ4P3L2ZNBQ6QQ, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.013927621 (569h6m25.885010666s) [noise=true , address=45.91.116.9:4264 , piece_id=GTJKAG5VAQXIY3DUSWC37ASCS6GGUC4HV6SHSQCDHOOHXVTIRBCA, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.015007457 (569h6m25.886090503s) [noise=true , address=145.79.227.16:4910 , piece_id=JWG3LVMM2QGC5YHUMBOGG45LERASIIWQVLWGPJMNKRRSYT3M7OYA, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.015304422 (569h6m25.886387457s) [noise=true , address=45.91.116.8:4045 , piece_id=JW5GV6RDF4TSDYCGK335V4U7FWZ35ZSOVR2ZTX4GGS2VGWJU4NCA, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.016535587 (569h6m25.887618632s) [noise=true , address=45.91.116.5:4299 , piece_id=NBHMOKJZH5TN543EZCYWZIFZ3LYKRJ3BSRLOEMG4INJ5VS7K63KQ, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.017629620 (569h6m25.888712655s) [noise=true , address=145.79.227.19:4077 , piece_id=DMXTHNHIMWQYKZCP4CY4MWS4GHAHH5PWVXI7Q73E6C5HLQP6JSSQ, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.018767277 (569h6m25.889850312s) [noise=true , address=145.79.227.48:4302 , piece_id=WPINK2W3URTINL2KXID5JDIMIPJBOAWDPYBNCVVU6DRVB2QMBO4Q, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.019844548 (569h6m25.890927593s) [noise=true , address=45.91.116.13:4156 , piece_id=2AERQMVUL7JIIL6LPSYMZHG7WSNZCORB5IUPCJ5K6K5MVJ6XOBPA, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.019941272 (569h6m25.891024307s) [noise=true , address=45.91.116.14:4190 , piece_id=6GFGFQ256LTLOUXMOERJMIBLPW3WTM3NRWA4BT6JLH5NJ54L2O4Q, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.021027521 (569h6m25.892110566s) [noise=true , address=145.79.227.11:4730 , piece_id=3GFJ5HEXCQR24DVL3QQDOBNPM52XRTJOYQSGC7YEZXIPPQHFNXIQ, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.021216440 (569h6m25.892299475s) [noise=true , address=145.79.227.49:4337 , piece_id=3WGEPNMRHQ4WCZYCE7JLUIJEZZ6HVDGA7EQBK2CSQ6FXIN6Z3BYA, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:26.022330972 (569h6m25.893414017s) [noise=true , address=45.91.116.11:4120 , piece_id=6QYPMT6UEB7ZTUG2EY3CSQY37KDBOVX7NMDOC4SBMVR2NPBEK2BQ, seg_pos={0 0}, upload=498956]: Uploading piece... Dec 29 08:35:27.705558259 (569h6m27.576641304s) [upload=498956]: Next returned nil segment Dec 29 08:35:27.705576423 (569h6m27.576659458s) [upload=498956]: Waiting for error group managing segments... Dec 29 08:35:27.997360655 (569h6m27.86844369s) [noise=true , address=145.79.227.52:4447 , piece_id=SRAT6RFRJBE6WKPJERYMJQJD6W2UIUC5AB4JYZWTRVDHU7LV74PA, seg_pos={0 0}, upload=498956]: Done uploading piece. err: <nil> Dec 29 08:35:28.004807876 (569h6m27.875890931s) [noise=true , address=145.79.227.16:4910 , piece_id=JWG3LVMM2QGC5YHUMBOGG45LERASIIWQVLWGPJMNKRRSYT3M7OYA, seg_pos={0 0}, upload=498956]: Done uploading piece. err: <nil> Dec 29 08:35:28.025316346 (569h6m27.896399381s) [noise=true , address=145.79.227.48:4302 , piece_id=WPINK2W3URTINL2KXID5JDIMIPJBOAWDPYBNCVVU6DRVB2QMBO4Q, seg_pos={0 0}, upload=498956]: Done uploading piece. err: <nil> Dec 29 08:35:28.026523225 (569h6m27.89760626s) [noise=true , address=145.79.227.11:4730 , piece_id=3GFJ5HEXCQR24DVL3QQDOBNPM52XRTJOYQSGC7YEZXIPPQHFNXIQ, seg_pos={0 0}, upload=498956]: Done uploading piece. err: <nil> Dec 29 08:35:28.038168099 (569h6m27.909251144s) [noise=true , address=145.79.227.19:4077 , piece_id=DMXTHNHIMWQYKZCP4CY4MWS4GHAHH5PWVXI7Q73E6C5HLQP6JSSQ, seg_pos={0 0}, upload=498956]: Done uploading piece. err: <nil> Dec 29 08:35:28.053740540 (569h6m27.924823585s) [noise=true , address=145.79.227.6:4258 , piece_id=M5TXYMDXODXXNZLFUE4TENVUJERH6M6P3N2S6HFHWVLJMT3453AA, seg_pos={0 0}, upload=498956]: Done uploading piece. err: <nil> Dec 29 08:35:28.210002928 (569h6m28.081085963s) [noise=true , address=77.90.31.10:4517 , piece_id=IVC6UZ5E7XJ6JZCIJ725JJYDLZXTDEJS7FHYYETZ4P3L2ZNBQ6QQ, seg_pos={0 0}, upload=498956]: Done uploading piece. err: <nil> Dec 29 08:35:28.264562077 (569h6m28.135645112s) [noise=true , address=45.91.116.9:4264 , piece_id=GTJKAG5VAQXIY3DUSWC37ASCS6GGUC4HV6SHSQCDHOOHXVTIRBCA, seg_pos={0 0}, upload=498956]: Done uploading piece. err: <nil> Dec 29 08:35:28.272373861 (569h6m28.143456896s) [noise=true , address=45.91.116.11:4120 , piece_id=6QYPMT6UEB7ZTUG2EY3CSQY37KDBOVX7NMDOC4SBMVR2NPBEK2BQ, seg_pos={0 0}, upload=498956]: Done uploading piece. err: <nil> Dec 29 08:35:28.356661147 (569h6m28.227744202s) [noise=true , address=45.91.116.7:4010 , piece_id=CKKZO7ZTIGAQZ2NHULIGM5VZ76NLPUZJWLMKRWB34BDKRSRWGPIA, seg_pos={0 0}, upload=498956]: Done uploading piece. err: <nil> Dec 29 08:35:28.365782435 (569h6m28.23686547s) [noise=true , address=45.91.116.5:4299 , piece_id=NBHMOKJZH5TN543EZCYWZIFZ3LYKRJ3BSRLOEMG4INJ5VS7K63KQ, seg_pos={0 0}, upload=498956]: Done uploading piece. err: <nil> Dec 29 08:35:28.444213861 (569h6m28.315296906s) [noise=true , address=45.91.116.13:4156 , piece_id=2AERQMVUL7JIIL6LPSYMZHG7WSNZCORB5IUPCJ5K6K5MVJ6XOBPA, seg_pos={0 0}, upload=498956]: Done uploading piece. err: <nil> Dec 29 08:35:28.457104276 (569h6m28.328187351s) [noise=true , address=45.91.116.14:4190 , piece_id=6GFGFQ256LTLOUXMOERJMIBLPW3WTM3NRWA4BT6JLH5NJ54L2O4Q, seg_pos={0 0}, upload=498956]: Done uploading piece. err: <nil> Dec 29 08:35:28.457133382 (569h6m28.328216427s) [seg_pos={0 0}, upload=498956]: Segment reached optimal threshold of 13 pieces. Dec 29 08:35:28.457344904 (569h6m28.328427969s) [noise=true , address=45.91.116.8:4045 , piece_id=JW5GV6RDF4TSDYCGK335V4U7FWZ35ZSOVR2ZTX4GGS2VGWJU4NCA, seg_pos={0 0}, upload=498956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:36024->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:36024->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36024->45.91.116.8:4045: use of closed network connection Dec 29 08:35:28.457352378 (569h6m28.328435413s) [noise=true , address=45.91.116.8:4045 , piece_id=JW5GV6RDF4TSDYCGK335V4U7FWZ35ZSOVR2ZTX4GGS2VGWJU4NCA, seg_pos={0 0}, upload=498956]: Long tail cancellation due to optimal threshold Dec 29 08:35:28.457528494 (569h6m28.328611539s) [noise=true , address=145.79.227.49:4337 , piece_id=3WGEPNMRHQ4WCZYCE7JLUIJEZZ6HVDGA7EQBK2CSQ6FXIN6Z3BYA, seg_pos={0 0}, upload=498956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 29 08:35:28.457544004 (569h6m28.328627039s) [noise=true , address=145.79.227.49:4337 , piece_id=3WGEPNMRHQ4WCZYCE7JLUIJEZZ6HVDGA7EQBK2CSQ6FXIN6Z3BYA, seg_pos={0 0}, upload=498956]: Long tail cancellation due to optimal threshold Dec 29 08:35:28.457738013 (569h6m28.328821058s) [noise=true , address=77.90.31.19:4842 , piece_id=TAG2JA7BG2PEFZNSKMD3LYTU3FIIWILS46BRNZ6T22EIWMSEYZNQ, seg_pos={0 0}, upload=498956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: read tcp 188.166.225.170:44186->77.90.31.19:4842: use of closed network connection; context canceled Dec 29 08:35:28.457746508 (569h6m28.328829554s) [noise=true , address=77.90.31.19:4842 , piece_id=TAG2JA7BG2PEFZNSKMD3LYTU3FIIWILS46BRNZ6T22EIWMSEYZNQ, seg_pos={0 0}, upload=498956]: Long tail cancellation due to optimal threshold Dec 29 08:35:28.457765465 (569h6m28.3288485s) [seg_pos={0 0}, upload=498956]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 08:35:28.457879251 (569h6m28.328962286s) [upload=498956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:35:28.457884461 (569h6m28.328967496s) [upload=498956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:35:28.751199387 (569h6m28.622282423s) [upload=498956]: Done uploading segments... Dec 29 08:35:28.751221018 (569h6m28.622304063s) [upload=498956]: Upload finished. err: <nil> Duration: 5.701523298s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...........X..X......X..........X.X..X..X.......X...........] duration: 3.410337946s trace: https://jaeger.dev.storj.io/trace/4ea4b41656f8a3e1 FAIL (2025-12-29 08:37:58 UTC) orbiter warning: context deadline exceeded Duration: 3.410366599s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)