Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [........X.......................XX....X.............X.......] duration: 3.732061495s trace: https://jaeger.dev.storj.io/trace/35fc14f889b57deb FAIL (2025-12-29 09:17:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.732088488s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.........X.......................XX....X.............X......] duration: 3.531894154s trace: https://jaeger.dev.storj.io/trace/35fc14f889b57deb FAIL (2025-12-29 09:17:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.531922298s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 15.306417414s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 09:17:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 15.306477418s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [......X....X..XXX..X.....X..XX.X...X...XX.X.XX..XXXX..X.....] duration: 5.608308901s trace: https://jaeger.dev.storj.io/trace/36e37e344142e911 FAIL (2025-12-29 09:16:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: JFE6CW3QI26KBIXNS52XY6Y32DGYB2SMEFPLRL2MLSLWCU5WFGVA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: LMUZWTP5EAMIKHH7DVOWZQEXYUDKWRQ67BZ5QPC2MJF72YNYQ2AA) context deadline exceeded ======LOGS====== Dec 29 09:13:44.873699497 (569h44m44.744782532s) [upload=499505]: Starting upload Dec 29 09:13:44.873751726 (569h44m44.744834761s) [upload=499505]: Done starting upload Dec 29 09:13:44.873959902 (569h44m44.745042937s) [upload=499505]: Uploading segments... Dec 29 09:13:44.873998657 (569h44m44.745081692s) [upload=499505]: Got next segment. Inline: false Dec 29 09:13:44.874021700 (569h44m44.745104735s) [upload=499505]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:13:44.874026048 (569h44m44.745109084s) [upload=499505]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:13:45.118775289 (569h44m44.989858324s) [seg_pos={0 0}, upload=499505]: Begin upload segment... Dec 29 09:13:45.118821938 (569h44m44.989904973s) [seg_pos={0 0}, upload=499505]: Done begin upload segment. Dec 29 09:13:45.118895308 (569h44m44.989978353s) [noise=true , address=45.91.116.14:4190 , piece_id=JFE6CW3QI26KBIXNS52XY6Y32DGYB2SMEFPLRL2MLSLWCU5WFGVA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.119220547 (569h44m44.990303593s) [noise=true , address=145.79.227.49:4338 , piece_id=ANJJO7K52EL5XHCJQ5YXWN3PF6XJCJLITZ7HTW4IWJ7UFK4RIYWQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.119303475 (569h44m44.99038651s) [noise=true , address=145.79.227.45:4230 , piece_id=MM2BZQFTJATCGCVGVDLZJ4WHMPF6UAPLCFHA2NSOXMGDYETYCEWA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.119928295 (569h44m44.99101134s) [noise=true , address=145.79.227.52:4447 , piece_id=ZXP46EA4VMCENZUYJ6TYOGCWQ5ESJXGELM45IBEFWE7JNMHQUVLQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.120609763 (569h44m44.991692798s) [noise=true , address=45.91.116.9:4261 , piece_id=LMUZWTP5EAMIKHH7DVOWZQEXYUDKWRQ67BZ5QPC2MJF72YNYQ2AA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.121166202 (569h44m44.992249238s) [noise=true , address=145.79.227.51:4410 , piece_id=MAEXGPXSO3X2HCK3I6TZAHYKTGTU2OIWTHZAKCONF2RGO7SLAXHQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.121579408 (569h44m44.992662443s) [noise=true , address=145.79.227.41:4086 , piece_id=I6FIZXJZFPLOM4ZE3X4NAN4XKEFFW2HRQZXQAFSXTZJAVGMZ46NA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.121915829 (569h44m44.992998874s) [noise=true , address=145.79.227.44:4193 , piece_id=6Q2X63UQMN4LZZEJNR5DBRZTDEZNZQMCJZR5P6LNLXAK75JRZF6A, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.122221782 (569h44m44.993304817s) [noise=true , address=145.79.227.25:4619 , piece_id=U3FDPWHYIWOVEEPFZSGNQSJS6PEU5SHF5EEJ557SWYKVZA5HF4QQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.122347471 (569h44m44.993430506s) [noise=true , address=45.91.116.7:4010 , piece_id=47EEPQ6DYJXEDC4SBH754GXANR3SB5TPJM2V5UURQB4W2U55N6QA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.122631302 (569h44m44.993714337s) [noise=true , address=145.79.227.15:4876 , piece_id=JU44HVQTRIFQ3AN4A2CEUSB2IMSKX22QKLMQQG7PGINA7TXCPFHA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.122659946 (569h44m44.993742981s) [noise=true , address=145.79.227.6:4258 , piece_id=UXUDQ7LHMWDX3ZL5ZE5AZYBNLPZOCXDGBNPB2YOTXVXBEMKKEYBQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.123160349 (569h44m44.994243394s) [noise=true , address=77.90.31.12:4589 , piece_id=67S2V6GRLTPKRIQ43UJOLW7WJKFSU36JZK3BTUGLE6CUSQS7GCSA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.123848570 (569h44m44.994931625s) [noise=true , address=145.79.227.11:4731 , piece_id=Y4DBCQ6JLJ5J3QXVNSYJ7RHVQVHKMVBUCATRWJSVMF73N37FC2TQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.123910607 (569h44m44.994993642s) [noise=true , address=145.79.227.26:4655 , piece_id=I5RYYWQIPV5FRJVPMMS7AMKZI2JXMLZSAPXQEHK3O4BI2E6QE2GQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.124656218 (569h44m44.995739253s) [noise=true , address=145.79.227.30:4762 , piece_id=5GQXZ4OB4NWJFRTCKWQCVNWXOZ2KIEAD56RHWBDC2ZLLAWDAV7AQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:46.835334850 (569h44m46.706417905s) [upload=499505]: Next returned nil segment Dec 29 09:13:46.835360229 (569h44m46.706443264s) [upload=499505]: Waiting for error group managing segments... Dec 29 09:13:47.109839540 (569h44m46.980922585s) [noise=true , address=145.79.227.15:4876 , piece_id=JU44HVQTRIFQ3AN4A2CEUSB2IMSKX22QKLMQQG7PGINA7TXCPFHA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.110411438 (569h44m46.981494483s) [noise=true , address=145.79.227.11:4731 , piece_id=Y4DBCQ6JLJ5J3QXVNSYJ7RHVQVHKMVBUCATRWJSVMF73N37FC2TQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.114183481 (569h44m46.985266526s) [noise=true , address=145.79.227.52:4447 , piece_id=ZXP46EA4VMCENZUYJ6TYOGCWQ5ESJXGELM45IBEFWE7JNMHQUVLQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.114435100 (569h44m46.985518136s) [noise=true , address=145.79.227.45:4230 , piece_id=MM2BZQFTJATCGCVGVDLZJ4WHMPF6UAPLCFHA2NSOXMGDYETYCEWA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.122670862 (569h44m46.993753907s) [noise=true , address=145.79.227.41:4086 , piece_id=I6FIZXJZFPLOM4ZE3X4NAN4XKEFFW2HRQZXQAFSXTZJAVGMZ46NA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.129831077 (569h44m47.000914132s) [noise=true , address=145.79.227.25:4619 , piece_id=U3FDPWHYIWOVEEPFZSGNQSJS6PEU5SHF5EEJ557SWYKVZA5HF4QQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.131201766 (569h44m47.002284812s) [noise=true , address=145.79.227.6:4258 , piece_id=UXUDQ7LHMWDX3ZL5ZE5AZYBNLPZOCXDGBNPB2YOTXVXBEMKKEYBQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.132685763 (569h44m47.003768808s) [noise=true , address=145.79.227.51:4410 , piece_id=MAEXGPXSO3X2HCK3I6TZAHYKTGTU2OIWTHZAKCONF2RGO7SLAXHQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.134482052 (569h44m47.005565087s) [noise=true , address=145.79.227.26:4655 , piece_id=I5RYYWQIPV5FRJVPMMS7AMKZI2JXMLZSAPXQEHK3O4BI2E6QE2GQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.472679973 (569h44m47.343763018s) [noise=true , address=45.91.116.7:4010 , piece_id=47EEPQ6DYJXEDC4SBH754GXANR3SB5TPJM2V5UURQB4W2U55N6QA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.509758196 (569h44m47.380841231s) [noise=true , address=45.91.116.14:4190 , piece_id=JFE6CW3QI26KBIXNS52XY6Y32DGYB2SMEFPLRL2MLSLWCU5WFGVA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.957149808 (569h44m47.828232843s) [noise=true , address=77.90.31.12:4589 , piece_id=67S2V6GRLTPKRIQ43UJOLW7WJKFSU36JZK3BTUGLE6CUSQS7GCSA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:48.056078764 (569h44m47.927161799s) [noise=true , address=45.91.116.9:4261 , piece_id=LMUZWTP5EAMIKHH7DVOWZQEXYUDKWRQ67BZ5QPC2MJF72YNYQ2AA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:48.056097490 (569h44m47.927180525s) [seg_pos={0 0}, upload=499505]: Segment reached optimal threshold of 13 pieces. Dec 29 09:13:48.056250281 (569h44m47.927333316s) [noise=true , address=145.79.227.49:4338 , piece_id=ANJJO7K52EL5XHCJQ5YXWN3PF6XJCJLITZ7HTW4IWJ7UFK4RIYWQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 29 09:13:48.056255992 (569h44m47.927339027s) [noise=true , address=145.79.227.49:4338 , piece_id=ANJJO7K52EL5XHCJQ5YXWN3PF6XJCJLITZ7HTW4IWJ7UFK4RIYWQ, seg_pos={0 0}, upload=499505]: Long tail cancellation due to optimal threshold Dec 29 09:13:48.056423150 (569h44m47.927506205s) [noise=true , address=145.79.227.44:4193 , piece_id=6Q2X63UQMN4LZZEJNR5DBRZTDEZNZQMCJZR5P6LNLXAK75JRZF6A, seg_pos={0 0}, upload=499505]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 29 09:13:48.056434101 (569h44m47.927517146s) [noise=true , address=145.79.227.44:4193 , piece_id=6Q2X63UQMN4LZZEJNR5DBRZTDEZNZQMCJZR5P6LNLXAK75JRZF6A, seg_pos={0 0}, upload=499505]: Long tail cancellation due to optimal threshold Dec 29 09:13:48.056583095 (569h44m47.92766613s) [noise=true , address=145.79.227.30:4762 , piece_id=5GQXZ4OB4NWJFRTCKWQCVNWXOZ2KIEAD56RHWBDC2ZLLAWDAV7AQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 09:13:48.056590429 (569h44m47.927673464s) [noise=true , address=145.79.227.30:4762 , piece_id=5GQXZ4OB4NWJFRTCKWQCVNWXOZ2KIEAD56RHWBDC2ZLLAWDAV7AQ, seg_pos={0 0}, upload=499505]: Long tail cancellation due to optimal threshold Dec 29 09:13:48.056608052 (569h44m47.927691087s) [seg_pos={0 0}, upload=499505]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:13:48.056673857 (569h44m47.927756892s) [upload=499505]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:13:48.056679157 (569h44m47.927762192s) [upload=499505]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:13:48.354466731 (569h44m48.225549786s) [upload=499505]: Done uploading segments... Dec 29 09:13:48.354490666 (569h44m48.225573711s) [upload=499505]: Upload finished. err: <nil> Duration: 5.6083394s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 42.598724685s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 09:16:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 42.598771544s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..........X.......................XX....X.............X.....] duration: 3.565503963s trace: https://jaeger.dev.storj.io/trace/35fc14f889b57deb FAIL (2025-12-29 09:16:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.565530114s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X......X....X..XXX..X.....X..XX.X...X...XX.X.XX..XXXX..X....] duration: 6.033198278s trace: https://jaeger.dev.storj.io/trace/36e37e344142e911 FAIL (2025-12-29 09:16:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: JFE6CW3QI26KBIXNS52XY6Y32DGYB2SMEFPLRL2MLSLWCU5WFGVA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: LMUZWTP5EAMIKHH7DVOWZQEXYUDKWRQ67BZ5QPC2MJF72YNYQ2AA) context deadline exceeded ======LOGS====== Dec 29 09:13:44.873699497 (569h44m44.744782532s) [upload=499505]: Starting upload Dec 29 09:13:44.873751726 (569h44m44.744834761s) [upload=499505]: Done starting upload Dec 29 09:13:44.873959902 (569h44m44.745042937s) [upload=499505]: Uploading segments... Dec 29 09:13:44.873998657 (569h44m44.745081692s) [upload=499505]: Got next segment. Inline: false Dec 29 09:13:44.874021700 (569h44m44.745104735s) [upload=499505]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:13:44.874026048 (569h44m44.745109084s) [upload=499505]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:13:45.118775289 (569h44m44.989858324s) [seg_pos={0 0}, upload=499505]: Begin upload segment... Dec 29 09:13:45.118821938 (569h44m44.989904973s) [seg_pos={0 0}, upload=499505]: Done begin upload segment. Dec 29 09:13:45.118895308 (569h44m44.989978353s) [noise=true , address=45.91.116.14:4190 , piece_id=JFE6CW3QI26KBIXNS52XY6Y32DGYB2SMEFPLRL2MLSLWCU5WFGVA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.119220547 (569h44m44.990303593s) [noise=true , address=145.79.227.49:4338 , piece_id=ANJJO7K52EL5XHCJQ5YXWN3PF6XJCJLITZ7HTW4IWJ7UFK4RIYWQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.119303475 (569h44m44.99038651s) [noise=true , address=145.79.227.45:4230 , piece_id=MM2BZQFTJATCGCVGVDLZJ4WHMPF6UAPLCFHA2NSOXMGDYETYCEWA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.119928295 (569h44m44.99101134s) [noise=true , address=145.79.227.52:4447 , piece_id=ZXP46EA4VMCENZUYJ6TYOGCWQ5ESJXGELM45IBEFWE7JNMHQUVLQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.120609763 (569h44m44.991692798s) [noise=true , address=45.91.116.9:4261 , piece_id=LMUZWTP5EAMIKHH7DVOWZQEXYUDKWRQ67BZ5QPC2MJF72YNYQ2AA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.121166202 (569h44m44.992249238s) [noise=true , address=145.79.227.51:4410 , piece_id=MAEXGPXSO3X2HCK3I6TZAHYKTGTU2OIWTHZAKCONF2RGO7SLAXHQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.121579408 (569h44m44.992662443s) [noise=true , address=145.79.227.41:4086 , piece_id=I6FIZXJZFPLOM4ZE3X4NAN4XKEFFW2HRQZXQAFSXTZJAVGMZ46NA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.121915829 (569h44m44.992998874s) [noise=true , address=145.79.227.44:4193 , piece_id=6Q2X63UQMN4LZZEJNR5DBRZTDEZNZQMCJZR5P6LNLXAK75JRZF6A, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.122221782 (569h44m44.993304817s) [noise=true , address=145.79.227.25:4619 , piece_id=U3FDPWHYIWOVEEPFZSGNQSJS6PEU5SHF5EEJ557SWYKVZA5HF4QQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.122347471 (569h44m44.993430506s) [noise=true , address=45.91.116.7:4010 , piece_id=47EEPQ6DYJXEDC4SBH754GXANR3SB5TPJM2V5UURQB4W2U55N6QA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.122631302 (569h44m44.993714337s) [noise=true , address=145.79.227.15:4876 , piece_id=JU44HVQTRIFQ3AN4A2CEUSB2IMSKX22QKLMQQG7PGINA7TXCPFHA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.122659946 (569h44m44.993742981s) [noise=true , address=145.79.227.6:4258 , piece_id=UXUDQ7LHMWDX3ZL5ZE5AZYBNLPZOCXDGBNPB2YOTXVXBEMKKEYBQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.123160349 (569h44m44.994243394s) [noise=true , address=77.90.31.12:4589 , piece_id=67S2V6GRLTPKRIQ43UJOLW7WJKFSU36JZK3BTUGLE6CUSQS7GCSA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.123848570 (569h44m44.994931625s) [noise=true , address=145.79.227.11:4731 , piece_id=Y4DBCQ6JLJ5J3QXVNSYJ7RHVQVHKMVBUCATRWJSVMF73N37FC2TQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.123910607 (569h44m44.994993642s) [noise=true , address=145.79.227.26:4655 , piece_id=I5RYYWQIPV5FRJVPMMS7AMKZI2JXMLZSAPXQEHK3O4BI2E6QE2GQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.124656218 (569h44m44.995739253s) [noise=true , address=145.79.227.30:4762 , piece_id=5GQXZ4OB4NWJFRTCKWQCVNWXOZ2KIEAD56RHWBDC2ZLLAWDAV7AQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:46.835334850 (569h44m46.706417905s) [upload=499505]: Next returned nil segment Dec 29 09:13:46.835360229 (569h44m46.706443264s) [upload=499505]: Waiting for error group managing segments... Dec 29 09:13:47.109839540 (569h44m46.980922585s) [noise=true , address=145.79.227.15:4876 , piece_id=JU44HVQTRIFQ3AN4A2CEUSB2IMSKX22QKLMQQG7PGINA7TXCPFHA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.110411438 (569h44m46.981494483s) [noise=true , address=145.79.227.11:4731 , piece_id=Y4DBCQ6JLJ5J3QXVNSYJ7RHVQVHKMVBUCATRWJSVMF73N37FC2TQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.114183481 (569h44m46.985266526s) [noise=true , address=145.79.227.52:4447 , piece_id=ZXP46EA4VMCENZUYJ6TYOGCWQ5ESJXGELM45IBEFWE7JNMHQUVLQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.114435100 (569h44m46.985518136s) [noise=true , address=145.79.227.45:4230 , piece_id=MM2BZQFTJATCGCVGVDLZJ4WHMPF6UAPLCFHA2NSOXMGDYETYCEWA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.122670862 (569h44m46.993753907s) [noise=true , address=145.79.227.41:4086 , piece_id=I6FIZXJZFPLOM4ZE3X4NAN4XKEFFW2HRQZXQAFSXTZJAVGMZ46NA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.129831077 (569h44m47.000914132s) [noise=true , address=145.79.227.25:4619 , piece_id=U3FDPWHYIWOVEEPFZSGNQSJS6PEU5SHF5EEJ557SWYKVZA5HF4QQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.131201766 (569h44m47.002284812s) [noise=true , address=145.79.227.6:4258 , piece_id=UXUDQ7LHMWDX3ZL5ZE5AZYBNLPZOCXDGBNPB2YOTXVXBEMKKEYBQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.132685763 (569h44m47.003768808s) [noise=true , address=145.79.227.51:4410 , piece_id=MAEXGPXSO3X2HCK3I6TZAHYKTGTU2OIWTHZAKCONF2RGO7SLAXHQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.134482052 (569h44m47.005565087s) [noise=true , address=145.79.227.26:4655 , piece_id=I5RYYWQIPV5FRJVPMMS7AMKZI2JXMLZSAPXQEHK3O4BI2E6QE2GQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.472679973 (569h44m47.343763018s) [noise=true , address=45.91.116.7:4010 , piece_id=47EEPQ6DYJXEDC4SBH754GXANR3SB5TPJM2V5UURQB4W2U55N6QA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.509758196 (569h44m47.380841231s) [noise=true , address=45.91.116.14:4190 , piece_id=JFE6CW3QI26KBIXNS52XY6Y32DGYB2SMEFPLRL2MLSLWCU5WFGVA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.957149808 (569h44m47.828232843s) [noise=true , address=77.90.31.12:4589 , piece_id=67S2V6GRLTPKRIQ43UJOLW7WJKFSU36JZK3BTUGLE6CUSQS7GCSA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:48.056078764 (569h44m47.927161799s) [noise=true , address=45.91.116.9:4261 , piece_id=LMUZWTP5EAMIKHH7DVOWZQEXYUDKWRQ67BZ5QPC2MJF72YNYQ2AA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:48.056097490 (569h44m47.927180525s) [seg_pos={0 0}, upload=499505]: Segment reached optimal threshold of 13 pieces. Dec 29 09:13:48.056250281 (569h44m47.927333316s) [noise=true , address=145.79.227.49:4338 , piece_id=ANJJO7K52EL5XHCJQ5YXWN3PF6XJCJLITZ7HTW4IWJ7UFK4RIYWQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 29 09:13:48.056255992 (569h44m47.927339027s) [noise=true , address=145.79.227.49:4338 , piece_id=ANJJO7K52EL5XHCJQ5YXWN3PF6XJCJLITZ7HTW4IWJ7UFK4RIYWQ, seg_pos={0 0}, upload=499505]: Long tail cancellation due to optimal threshold Dec 29 09:13:48.056423150 (569h44m47.927506205s) [noise=true , address=145.79.227.44:4193 , piece_id=6Q2X63UQMN4LZZEJNR5DBRZTDEZNZQMCJZR5P6LNLXAK75JRZF6A, seg_pos={0 0}, upload=499505]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 29 09:13:48.056434101 (569h44m47.927517146s) [noise=true , address=145.79.227.44:4193 , piece_id=6Q2X63UQMN4LZZEJNR5DBRZTDEZNZQMCJZR5P6LNLXAK75JRZF6A, seg_pos={0 0}, upload=499505]: Long tail cancellation due to optimal threshold Dec 29 09:13:48.056583095 (569h44m47.92766613s) [noise=true , address=145.79.227.30:4762 , piece_id=5GQXZ4OB4NWJFRTCKWQCVNWXOZ2KIEAD56RHWBDC2ZLLAWDAV7AQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 09:13:48.056590429 (569h44m47.927673464s) [noise=true , address=145.79.227.30:4762 , piece_id=5GQXZ4OB4NWJFRTCKWQCVNWXOZ2KIEAD56RHWBDC2ZLLAWDAV7AQ, seg_pos={0 0}, upload=499505]: Long tail cancellation due to optimal threshold Dec 29 09:13:48.056608052 (569h44m47.927691087s) [seg_pos={0 0}, upload=499505]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:13:48.056673857 (569h44m47.927756892s) [upload=499505]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:13:48.056679157 (569h44m47.927762192s) [upload=499505]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:13:48.354466731 (569h44m48.225549786s) [upload=499505]: Done uploading segments... Dec 29 09:13:48.354490666 (569h44m48.225573711s) [upload=499505]: Upload finished. err: <nil> Duration: 6.033224858s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...........X.......................XX....X.............X....] duration: 3.646206146s trace: https://jaeger.dev.storj.io/trace/35fc14f889b57deb FAIL (2025-12-29 09:15:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.646233549s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XX......X....X..XXX..X.....X..XX.X...X...XX.X.XX..XXXX..X...] duration: 5.999012343s trace: https://jaeger.dev.storj.io/trace/36e37e344142e911 FAIL (2025-12-29 09:15:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: JFE6CW3QI26KBIXNS52XY6Y32DGYB2SMEFPLRL2MLSLWCU5WFGVA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: LMUZWTP5EAMIKHH7DVOWZQEXYUDKWRQ67BZ5QPC2MJF72YNYQ2AA) context deadline exceeded ======LOGS====== Dec 29 09:13:44.873699497 (569h44m44.744782532s) [upload=499505]: Starting upload Dec 29 09:13:44.873751726 (569h44m44.744834761s) [upload=499505]: Done starting upload Dec 29 09:13:44.873959902 (569h44m44.745042937s) [upload=499505]: Uploading segments... Dec 29 09:13:44.873998657 (569h44m44.745081692s) [upload=499505]: Got next segment. Inline: false Dec 29 09:13:44.874021700 (569h44m44.745104735s) [upload=499505]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:13:44.874026048 (569h44m44.745109084s) [upload=499505]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:13:45.118775289 (569h44m44.989858324s) [seg_pos={0 0}, upload=499505]: Begin upload segment... Dec 29 09:13:45.118821938 (569h44m44.989904973s) [seg_pos={0 0}, upload=499505]: Done begin upload segment. Dec 29 09:13:45.118895308 (569h44m44.989978353s) [noise=true , address=45.91.116.14:4190 , piece_id=JFE6CW3QI26KBIXNS52XY6Y32DGYB2SMEFPLRL2MLSLWCU5WFGVA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.119220547 (569h44m44.990303593s) [noise=true , address=145.79.227.49:4338 , piece_id=ANJJO7K52EL5XHCJQ5YXWN3PF6XJCJLITZ7HTW4IWJ7UFK4RIYWQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.119303475 (569h44m44.99038651s) [noise=true , address=145.79.227.45:4230 , piece_id=MM2BZQFTJATCGCVGVDLZJ4WHMPF6UAPLCFHA2NSOXMGDYETYCEWA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.119928295 (569h44m44.99101134s) [noise=true , address=145.79.227.52:4447 , piece_id=ZXP46EA4VMCENZUYJ6TYOGCWQ5ESJXGELM45IBEFWE7JNMHQUVLQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.120609763 (569h44m44.991692798s) [noise=true , address=45.91.116.9:4261 , piece_id=LMUZWTP5EAMIKHH7DVOWZQEXYUDKWRQ67BZ5QPC2MJF72YNYQ2AA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.121166202 (569h44m44.992249238s) [noise=true , address=145.79.227.51:4410 , piece_id=MAEXGPXSO3X2HCK3I6TZAHYKTGTU2OIWTHZAKCONF2RGO7SLAXHQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.121579408 (569h44m44.992662443s) [noise=true , address=145.79.227.41:4086 , piece_id=I6FIZXJZFPLOM4ZE3X4NAN4XKEFFW2HRQZXQAFSXTZJAVGMZ46NA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.121915829 (569h44m44.992998874s) [noise=true , address=145.79.227.44:4193 , piece_id=6Q2X63UQMN4LZZEJNR5DBRZTDEZNZQMCJZR5P6LNLXAK75JRZF6A, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.122221782 (569h44m44.993304817s) [noise=true , address=145.79.227.25:4619 , piece_id=U3FDPWHYIWOVEEPFZSGNQSJS6PEU5SHF5EEJ557SWYKVZA5HF4QQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.122347471 (569h44m44.993430506s) [noise=true , address=45.91.116.7:4010 , piece_id=47EEPQ6DYJXEDC4SBH754GXANR3SB5TPJM2V5UURQB4W2U55N6QA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.122631302 (569h44m44.993714337s) [noise=true , address=145.79.227.15:4876 , piece_id=JU44HVQTRIFQ3AN4A2CEUSB2IMSKX22QKLMQQG7PGINA7TXCPFHA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.122659946 (569h44m44.993742981s) [noise=true , address=145.79.227.6:4258 , piece_id=UXUDQ7LHMWDX3ZL5ZE5AZYBNLPZOCXDGBNPB2YOTXVXBEMKKEYBQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.123160349 (569h44m44.994243394s) [noise=true , address=77.90.31.12:4589 , piece_id=67S2V6GRLTPKRIQ43UJOLW7WJKFSU36JZK3BTUGLE6CUSQS7GCSA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.123848570 (569h44m44.994931625s) [noise=true , address=145.79.227.11:4731 , piece_id=Y4DBCQ6JLJ5J3QXVNSYJ7RHVQVHKMVBUCATRWJSVMF73N37FC2TQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.123910607 (569h44m44.994993642s) [noise=true , address=145.79.227.26:4655 , piece_id=I5RYYWQIPV5FRJVPMMS7AMKZI2JXMLZSAPXQEHK3O4BI2E6QE2GQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.124656218 (569h44m44.995739253s) [noise=true , address=145.79.227.30:4762 , piece_id=5GQXZ4OB4NWJFRTCKWQCVNWXOZ2KIEAD56RHWBDC2ZLLAWDAV7AQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:46.835334850 (569h44m46.706417905s) [upload=499505]: Next returned nil segment Dec 29 09:13:46.835360229 (569h44m46.706443264s) [upload=499505]: Waiting for error group managing segments... Dec 29 09:13:47.109839540 (569h44m46.980922585s) [noise=true , address=145.79.227.15:4876 , piece_id=JU44HVQTRIFQ3AN4A2CEUSB2IMSKX22QKLMQQG7PGINA7TXCPFHA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.110411438 (569h44m46.981494483s) [noise=true , address=145.79.227.11:4731 , piece_id=Y4DBCQ6JLJ5J3QXVNSYJ7RHVQVHKMVBUCATRWJSVMF73N37FC2TQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.114183481 (569h44m46.985266526s) [noise=true , address=145.79.227.52:4447 , piece_id=ZXP46EA4VMCENZUYJ6TYOGCWQ5ESJXGELM45IBEFWE7JNMHQUVLQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.114435100 (569h44m46.985518136s) [noise=true , address=145.79.227.45:4230 , piece_id=MM2BZQFTJATCGCVGVDLZJ4WHMPF6UAPLCFHA2NSOXMGDYETYCEWA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.122670862 (569h44m46.993753907s) [noise=true , address=145.79.227.41:4086 , piece_id=I6FIZXJZFPLOM4ZE3X4NAN4XKEFFW2HRQZXQAFSXTZJAVGMZ46NA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.129831077 (569h44m47.000914132s) [noise=true , address=145.79.227.25:4619 , piece_id=U3FDPWHYIWOVEEPFZSGNQSJS6PEU5SHF5EEJ557SWYKVZA5HF4QQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.131201766 (569h44m47.002284812s) [noise=true , address=145.79.227.6:4258 , piece_id=UXUDQ7LHMWDX3ZL5ZE5AZYBNLPZOCXDGBNPB2YOTXVXBEMKKEYBQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.132685763 (569h44m47.003768808s) [noise=true , address=145.79.227.51:4410 , piece_id=MAEXGPXSO3X2HCK3I6TZAHYKTGTU2OIWTHZAKCONF2RGO7SLAXHQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.134482052 (569h44m47.005565087s) [noise=true , address=145.79.227.26:4655 , piece_id=I5RYYWQIPV5FRJVPMMS7AMKZI2JXMLZSAPXQEHK3O4BI2E6QE2GQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.472679973 (569h44m47.343763018s) [noise=true , address=45.91.116.7:4010 , piece_id=47EEPQ6DYJXEDC4SBH754GXANR3SB5TPJM2V5UURQB4W2U55N6QA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.509758196 (569h44m47.380841231s) [noise=true , address=45.91.116.14:4190 , piece_id=JFE6CW3QI26KBIXNS52XY6Y32DGYB2SMEFPLRL2MLSLWCU5WFGVA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.957149808 (569h44m47.828232843s) [noise=true , address=77.90.31.12:4589 , piece_id=67S2V6GRLTPKRIQ43UJOLW7WJKFSU36JZK3BTUGLE6CUSQS7GCSA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:48.056078764 (569h44m47.927161799s) [noise=true , address=45.91.116.9:4261 , piece_id=LMUZWTP5EAMIKHH7DVOWZQEXYUDKWRQ67BZ5QPC2MJF72YNYQ2AA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:48.056097490 (569h44m47.927180525s) [seg_pos={0 0}, upload=499505]: Segment reached optimal threshold of 13 pieces. Dec 29 09:13:48.056250281 (569h44m47.927333316s) [noise=true , address=145.79.227.49:4338 , piece_id=ANJJO7K52EL5XHCJQ5YXWN3PF6XJCJLITZ7HTW4IWJ7UFK4RIYWQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 29 09:13:48.056255992 (569h44m47.927339027s) [noise=true , address=145.79.227.49:4338 , piece_id=ANJJO7K52EL5XHCJQ5YXWN3PF6XJCJLITZ7HTW4IWJ7UFK4RIYWQ, seg_pos={0 0}, upload=499505]: Long tail cancellation due to optimal threshold Dec 29 09:13:48.056423150 (569h44m47.927506205s) [noise=true , address=145.79.227.44:4193 , piece_id=6Q2X63UQMN4LZZEJNR5DBRZTDEZNZQMCJZR5P6LNLXAK75JRZF6A, seg_pos={0 0}, upload=499505]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 29 09:13:48.056434101 (569h44m47.927517146s) [noise=true , address=145.79.227.44:4193 , piece_id=6Q2X63UQMN4LZZEJNR5DBRZTDEZNZQMCJZR5P6LNLXAK75JRZF6A, seg_pos={0 0}, upload=499505]: Long tail cancellation due to optimal threshold Dec 29 09:13:48.056583095 (569h44m47.92766613s) [noise=true , address=145.79.227.30:4762 , piece_id=5GQXZ4OB4NWJFRTCKWQCVNWXOZ2KIEAD56RHWBDC2ZLLAWDAV7AQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 09:13:48.056590429 (569h44m47.927673464s) [noise=true , address=145.79.227.30:4762 , piece_id=5GQXZ4OB4NWJFRTCKWQCVNWXOZ2KIEAD56RHWBDC2ZLLAWDAV7AQ, seg_pos={0 0}, upload=499505]: Long tail cancellation due to optimal threshold Dec 29 09:13:48.056608052 (569h44m47.927691087s) [seg_pos={0 0}, upload=499505]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:13:48.056673857 (569h44m47.927756892s) [upload=499505]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:13:48.056679157 (569h44m47.927762192s) [upload=499505]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:13:48.354466731 (569h44m48.225549786s) [upload=499505]: Done uploading segments... Dec 29 09:13:48.354490666 (569h44m48.225573711s) [upload=499505]: Upload finished. err: <nil> Duration: 5.999046528s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [............X.......................XX....X.............X...] duration: 4.156447576s trace: https://jaeger.dev.storj.io/trace/35fc14f889b57deb FAIL (2025-12-29 09:15:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.156478525s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 20.665590292s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 09:14:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 20.665615881s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXX......X....X..XXX..X.....X..XX.X...X...XX.X.XX..XXXX..X..] duration: 5.966530487s trace: https://jaeger.dev.storj.io/trace/36e37e344142e911 FAIL (2025-12-29 09:14:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: JFE6CW3QI26KBIXNS52XY6Y32DGYB2SMEFPLRL2MLSLWCU5WFGVA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: LMUZWTP5EAMIKHH7DVOWZQEXYUDKWRQ67BZ5QPC2MJF72YNYQ2AA) context deadline exceeded ======LOGS====== Dec 29 09:13:44.873699497 (569h44m44.744782532s) [upload=499505]: Starting upload Dec 29 09:13:44.873751726 (569h44m44.744834761s) [upload=499505]: Done starting upload Dec 29 09:13:44.873959902 (569h44m44.745042937s) [upload=499505]: Uploading segments... Dec 29 09:13:44.873998657 (569h44m44.745081692s) [upload=499505]: Got next segment. Inline: false Dec 29 09:13:44.874021700 (569h44m44.745104735s) [upload=499505]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:13:44.874026048 (569h44m44.745109084s) [upload=499505]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:13:45.118775289 (569h44m44.989858324s) [seg_pos={0 0}, upload=499505]: Begin upload segment... Dec 29 09:13:45.118821938 (569h44m44.989904973s) [seg_pos={0 0}, upload=499505]: Done begin upload segment. Dec 29 09:13:45.118895308 (569h44m44.989978353s) [noise=true , address=45.91.116.14:4190 , piece_id=JFE6CW3QI26KBIXNS52XY6Y32DGYB2SMEFPLRL2MLSLWCU5WFGVA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.119220547 (569h44m44.990303593s) [noise=true , address=145.79.227.49:4338 , piece_id=ANJJO7K52EL5XHCJQ5YXWN3PF6XJCJLITZ7HTW4IWJ7UFK4RIYWQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.119303475 (569h44m44.99038651s) [noise=true , address=145.79.227.45:4230 , piece_id=MM2BZQFTJATCGCVGVDLZJ4WHMPF6UAPLCFHA2NSOXMGDYETYCEWA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.119928295 (569h44m44.99101134s) [noise=true , address=145.79.227.52:4447 , piece_id=ZXP46EA4VMCENZUYJ6TYOGCWQ5ESJXGELM45IBEFWE7JNMHQUVLQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.120609763 (569h44m44.991692798s) [noise=true , address=45.91.116.9:4261 , piece_id=LMUZWTP5EAMIKHH7DVOWZQEXYUDKWRQ67BZ5QPC2MJF72YNYQ2AA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.121166202 (569h44m44.992249238s) [noise=true , address=145.79.227.51:4410 , piece_id=MAEXGPXSO3X2HCK3I6TZAHYKTGTU2OIWTHZAKCONF2RGO7SLAXHQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.121579408 (569h44m44.992662443s) [noise=true , address=145.79.227.41:4086 , piece_id=I6FIZXJZFPLOM4ZE3X4NAN4XKEFFW2HRQZXQAFSXTZJAVGMZ46NA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.121915829 (569h44m44.992998874s) [noise=true , address=145.79.227.44:4193 , piece_id=6Q2X63UQMN4LZZEJNR5DBRZTDEZNZQMCJZR5P6LNLXAK75JRZF6A, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.122221782 (569h44m44.993304817s) [noise=true , address=145.79.227.25:4619 , piece_id=U3FDPWHYIWOVEEPFZSGNQSJS6PEU5SHF5EEJ557SWYKVZA5HF4QQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.122347471 (569h44m44.993430506s) [noise=true , address=45.91.116.7:4010 , piece_id=47EEPQ6DYJXEDC4SBH754GXANR3SB5TPJM2V5UURQB4W2U55N6QA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.122631302 (569h44m44.993714337s) [noise=true , address=145.79.227.15:4876 , piece_id=JU44HVQTRIFQ3AN4A2CEUSB2IMSKX22QKLMQQG7PGINA7TXCPFHA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.122659946 (569h44m44.993742981s) [noise=true , address=145.79.227.6:4258 , piece_id=UXUDQ7LHMWDX3ZL5ZE5AZYBNLPZOCXDGBNPB2YOTXVXBEMKKEYBQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.123160349 (569h44m44.994243394s) [noise=true , address=77.90.31.12:4589 , piece_id=67S2V6GRLTPKRIQ43UJOLW7WJKFSU36JZK3BTUGLE6CUSQS7GCSA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.123848570 (569h44m44.994931625s) [noise=true , address=145.79.227.11:4731 , piece_id=Y4DBCQ6JLJ5J3QXVNSYJ7RHVQVHKMVBUCATRWJSVMF73N37FC2TQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.123910607 (569h44m44.994993642s) [noise=true , address=145.79.227.26:4655 , piece_id=I5RYYWQIPV5FRJVPMMS7AMKZI2JXMLZSAPXQEHK3O4BI2E6QE2GQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.124656218 (569h44m44.995739253s) [noise=true , address=145.79.227.30:4762 , piece_id=5GQXZ4OB4NWJFRTCKWQCVNWXOZ2KIEAD56RHWBDC2ZLLAWDAV7AQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:46.835334850 (569h44m46.706417905s) [upload=499505]: Next returned nil segment Dec 29 09:13:46.835360229 (569h44m46.706443264s) [upload=499505]: Waiting for error group managing segments... Dec 29 09:13:47.109839540 (569h44m46.980922585s) [noise=true , address=145.79.227.15:4876 , piece_id=JU44HVQTRIFQ3AN4A2CEUSB2IMSKX22QKLMQQG7PGINA7TXCPFHA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.110411438 (569h44m46.981494483s) [noise=true , address=145.79.227.11:4731 , piece_id=Y4DBCQ6JLJ5J3QXVNSYJ7RHVQVHKMVBUCATRWJSVMF73N37FC2TQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.114183481 (569h44m46.985266526s) [noise=true , address=145.79.227.52:4447 , piece_id=ZXP46EA4VMCENZUYJ6TYOGCWQ5ESJXGELM45IBEFWE7JNMHQUVLQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.114435100 (569h44m46.985518136s) [noise=true , address=145.79.227.45:4230 , piece_id=MM2BZQFTJATCGCVGVDLZJ4WHMPF6UAPLCFHA2NSOXMGDYETYCEWA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.122670862 (569h44m46.993753907s) [noise=true , address=145.79.227.41:4086 , piece_id=I6FIZXJZFPLOM4ZE3X4NAN4XKEFFW2HRQZXQAFSXTZJAVGMZ46NA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.129831077 (569h44m47.000914132s) [noise=true , address=145.79.227.25:4619 , piece_id=U3FDPWHYIWOVEEPFZSGNQSJS6PEU5SHF5EEJ557SWYKVZA5HF4QQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.131201766 (569h44m47.002284812s) [noise=true , address=145.79.227.6:4258 , piece_id=UXUDQ7LHMWDX3ZL5ZE5AZYBNLPZOCXDGBNPB2YOTXVXBEMKKEYBQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.132685763 (569h44m47.003768808s) [noise=true , address=145.79.227.51:4410 , piece_id=MAEXGPXSO3X2HCK3I6TZAHYKTGTU2OIWTHZAKCONF2RGO7SLAXHQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.134482052 (569h44m47.005565087s) [noise=true , address=145.79.227.26:4655 , piece_id=I5RYYWQIPV5FRJVPMMS7AMKZI2JXMLZSAPXQEHK3O4BI2E6QE2GQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.472679973 (569h44m47.343763018s) [noise=true , address=45.91.116.7:4010 , piece_id=47EEPQ6DYJXEDC4SBH754GXANR3SB5TPJM2V5UURQB4W2U55N6QA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.509758196 (569h44m47.380841231s) [noise=true , address=45.91.116.14:4190 , piece_id=JFE6CW3QI26KBIXNS52XY6Y32DGYB2SMEFPLRL2MLSLWCU5WFGVA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.957149808 (569h44m47.828232843s) [noise=true , address=77.90.31.12:4589 , piece_id=67S2V6GRLTPKRIQ43UJOLW7WJKFSU36JZK3BTUGLE6CUSQS7GCSA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:48.056078764 (569h44m47.927161799s) [noise=true , address=45.91.116.9:4261 , piece_id=LMUZWTP5EAMIKHH7DVOWZQEXYUDKWRQ67BZ5QPC2MJF72YNYQ2AA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:48.056097490 (569h44m47.927180525s) [seg_pos={0 0}, upload=499505]: Segment reached optimal threshold of 13 pieces. Dec 29 09:13:48.056250281 (569h44m47.927333316s) [noise=true , address=145.79.227.49:4338 , piece_id=ANJJO7K52EL5XHCJQ5YXWN3PF6XJCJLITZ7HTW4IWJ7UFK4RIYWQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 29 09:13:48.056255992 (569h44m47.927339027s) [noise=true , address=145.79.227.49:4338 , piece_id=ANJJO7K52EL5XHCJQ5YXWN3PF6XJCJLITZ7HTW4IWJ7UFK4RIYWQ, seg_pos={0 0}, upload=499505]: Long tail cancellation due to optimal threshold Dec 29 09:13:48.056423150 (569h44m47.927506205s) [noise=true , address=145.79.227.44:4193 , piece_id=6Q2X63UQMN4LZZEJNR5DBRZTDEZNZQMCJZR5P6LNLXAK75JRZF6A, seg_pos={0 0}, upload=499505]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 29 09:13:48.056434101 (569h44m47.927517146s) [noise=true , address=145.79.227.44:4193 , piece_id=6Q2X63UQMN4LZZEJNR5DBRZTDEZNZQMCJZR5P6LNLXAK75JRZF6A, seg_pos={0 0}, upload=499505]: Long tail cancellation due to optimal threshold Dec 29 09:13:48.056583095 (569h44m47.92766613s) [noise=true , address=145.79.227.30:4762 , piece_id=5GQXZ4OB4NWJFRTCKWQCVNWXOZ2KIEAD56RHWBDC2ZLLAWDAV7AQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 09:13:48.056590429 (569h44m47.927673464s) [noise=true , address=145.79.227.30:4762 , piece_id=5GQXZ4OB4NWJFRTCKWQCVNWXOZ2KIEAD56RHWBDC2ZLLAWDAV7AQ, seg_pos={0 0}, upload=499505]: Long tail cancellation due to optimal threshold Dec 29 09:13:48.056608052 (569h44m47.927691087s) [seg_pos={0 0}, upload=499505]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:13:48.056673857 (569h44m47.927756892s) [upload=499505]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:13:48.056679157 (569h44m47.927762192s) [upload=499505]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:13:48.354466731 (569h44m48.225549786s) [upload=499505]: Done uploading segments... Dec 29 09:13:48.354490666 (569h44m48.225573711s) [upload=499505]: Upload finished. err: <nil> Duration: 5.966557839s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.XXX......X....X..XXX..X.....X..XX.X...X...XX.X.XX..XXXX..X.] duration: 5.931320477s trace: https://jaeger.dev.storj.io/trace/36e37e344142e911 FAIL (2025-12-29 09:14:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: JFE6CW3QI26KBIXNS52XY6Y32DGYB2SMEFPLRL2MLSLWCU5WFGVA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: LMUZWTP5EAMIKHH7DVOWZQEXYUDKWRQ67BZ5QPC2MJF72YNYQ2AA) context deadline exceeded ======LOGS====== Dec 29 09:13:44.873699497 (569h44m44.744782532s) [upload=499505]: Starting upload Dec 29 09:13:44.873751726 (569h44m44.744834761s) [upload=499505]: Done starting upload Dec 29 09:13:44.873959902 (569h44m44.745042937s) [upload=499505]: Uploading segments... Dec 29 09:13:44.873998657 (569h44m44.745081692s) [upload=499505]: Got next segment. Inline: false Dec 29 09:13:44.874021700 (569h44m44.745104735s) [upload=499505]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:13:44.874026048 (569h44m44.745109084s) [upload=499505]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:13:45.118775289 (569h44m44.989858324s) [seg_pos={0 0}, upload=499505]: Begin upload segment... Dec 29 09:13:45.118821938 (569h44m44.989904973s) [seg_pos={0 0}, upload=499505]: Done begin upload segment. Dec 29 09:13:45.118895308 (569h44m44.989978353s) [noise=true , address=45.91.116.14:4190 , piece_id=JFE6CW3QI26KBIXNS52XY6Y32DGYB2SMEFPLRL2MLSLWCU5WFGVA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.119220547 (569h44m44.990303593s) [noise=true , address=145.79.227.49:4338 , piece_id=ANJJO7K52EL5XHCJQ5YXWN3PF6XJCJLITZ7HTW4IWJ7UFK4RIYWQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.119303475 (569h44m44.99038651s) [noise=true , address=145.79.227.45:4230 , piece_id=MM2BZQFTJATCGCVGVDLZJ4WHMPF6UAPLCFHA2NSOXMGDYETYCEWA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.119928295 (569h44m44.99101134s) [noise=true , address=145.79.227.52:4447 , piece_id=ZXP46EA4VMCENZUYJ6TYOGCWQ5ESJXGELM45IBEFWE7JNMHQUVLQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.120609763 (569h44m44.991692798s) [noise=true , address=45.91.116.9:4261 , piece_id=LMUZWTP5EAMIKHH7DVOWZQEXYUDKWRQ67BZ5QPC2MJF72YNYQ2AA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.121166202 (569h44m44.992249238s) [noise=true , address=145.79.227.51:4410 , piece_id=MAEXGPXSO3X2HCK3I6TZAHYKTGTU2OIWTHZAKCONF2RGO7SLAXHQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.121579408 (569h44m44.992662443s) [noise=true , address=145.79.227.41:4086 , piece_id=I6FIZXJZFPLOM4ZE3X4NAN4XKEFFW2HRQZXQAFSXTZJAVGMZ46NA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.121915829 (569h44m44.992998874s) [noise=true , address=145.79.227.44:4193 , piece_id=6Q2X63UQMN4LZZEJNR5DBRZTDEZNZQMCJZR5P6LNLXAK75JRZF6A, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.122221782 (569h44m44.993304817s) [noise=true , address=145.79.227.25:4619 , piece_id=U3FDPWHYIWOVEEPFZSGNQSJS6PEU5SHF5EEJ557SWYKVZA5HF4QQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.122347471 (569h44m44.993430506s) [noise=true , address=45.91.116.7:4010 , piece_id=47EEPQ6DYJXEDC4SBH754GXANR3SB5TPJM2V5UURQB4W2U55N6QA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.122631302 (569h44m44.993714337s) [noise=true , address=145.79.227.15:4876 , piece_id=JU44HVQTRIFQ3AN4A2CEUSB2IMSKX22QKLMQQG7PGINA7TXCPFHA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.122659946 (569h44m44.993742981s) [noise=true , address=145.79.227.6:4258 , piece_id=UXUDQ7LHMWDX3ZL5ZE5AZYBNLPZOCXDGBNPB2YOTXVXBEMKKEYBQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.123160349 (569h44m44.994243394s) [noise=true , address=77.90.31.12:4589 , piece_id=67S2V6GRLTPKRIQ43UJOLW7WJKFSU36JZK3BTUGLE6CUSQS7GCSA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.123848570 (569h44m44.994931625s) [noise=true , address=145.79.227.11:4731 , piece_id=Y4DBCQ6JLJ5J3QXVNSYJ7RHVQVHKMVBUCATRWJSVMF73N37FC2TQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.123910607 (569h44m44.994993642s) [noise=true , address=145.79.227.26:4655 , piece_id=I5RYYWQIPV5FRJVPMMS7AMKZI2JXMLZSAPXQEHK3O4BI2E6QE2GQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.124656218 (569h44m44.995739253s) [noise=true , address=145.79.227.30:4762 , piece_id=5GQXZ4OB4NWJFRTCKWQCVNWXOZ2KIEAD56RHWBDC2ZLLAWDAV7AQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:46.835334850 (569h44m46.706417905s) [upload=499505]: Next returned nil segment Dec 29 09:13:46.835360229 (569h44m46.706443264s) [upload=499505]: Waiting for error group managing segments... Dec 29 09:13:47.109839540 (569h44m46.980922585s) [noise=true , address=145.79.227.15:4876 , piece_id=JU44HVQTRIFQ3AN4A2CEUSB2IMSKX22QKLMQQG7PGINA7TXCPFHA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.110411438 (569h44m46.981494483s) [noise=true , address=145.79.227.11:4731 , piece_id=Y4DBCQ6JLJ5J3QXVNSYJ7RHVQVHKMVBUCATRWJSVMF73N37FC2TQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.114183481 (569h44m46.985266526s) [noise=true , address=145.79.227.52:4447 , piece_id=ZXP46EA4VMCENZUYJ6TYOGCWQ5ESJXGELM45IBEFWE7JNMHQUVLQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.114435100 (569h44m46.985518136s) [noise=true , address=145.79.227.45:4230 , piece_id=MM2BZQFTJATCGCVGVDLZJ4WHMPF6UAPLCFHA2NSOXMGDYETYCEWA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.122670862 (569h44m46.993753907s) [noise=true , address=145.79.227.41:4086 , piece_id=I6FIZXJZFPLOM4ZE3X4NAN4XKEFFW2HRQZXQAFSXTZJAVGMZ46NA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.129831077 (569h44m47.000914132s) [noise=true , address=145.79.227.25:4619 , piece_id=U3FDPWHYIWOVEEPFZSGNQSJS6PEU5SHF5EEJ557SWYKVZA5HF4QQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.131201766 (569h44m47.002284812s) [noise=true , address=145.79.227.6:4258 , piece_id=UXUDQ7LHMWDX3ZL5ZE5AZYBNLPZOCXDGBNPB2YOTXVXBEMKKEYBQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.132685763 (569h44m47.003768808s) [noise=true , address=145.79.227.51:4410 , piece_id=MAEXGPXSO3X2HCK3I6TZAHYKTGTU2OIWTHZAKCONF2RGO7SLAXHQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.134482052 (569h44m47.005565087s) [noise=true , address=145.79.227.26:4655 , piece_id=I5RYYWQIPV5FRJVPMMS7AMKZI2JXMLZSAPXQEHK3O4BI2E6QE2GQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.472679973 (569h44m47.343763018s) [noise=true , address=45.91.116.7:4010 , piece_id=47EEPQ6DYJXEDC4SBH754GXANR3SB5TPJM2V5UURQB4W2U55N6QA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.509758196 (569h44m47.380841231s) [noise=true , address=45.91.116.14:4190 , piece_id=JFE6CW3QI26KBIXNS52XY6Y32DGYB2SMEFPLRL2MLSLWCU5WFGVA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.957149808 (569h44m47.828232843s) [noise=true , address=77.90.31.12:4589 , piece_id=67S2V6GRLTPKRIQ43UJOLW7WJKFSU36JZK3BTUGLE6CUSQS7GCSA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:48.056078764 (569h44m47.927161799s) [noise=true , address=45.91.116.9:4261 , piece_id=LMUZWTP5EAMIKHH7DVOWZQEXYUDKWRQ67BZ5QPC2MJF72YNYQ2AA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:48.056097490 (569h44m47.927180525s) [seg_pos={0 0}, upload=499505]: Segment reached optimal threshold of 13 pieces. Dec 29 09:13:48.056250281 (569h44m47.927333316s) [noise=true , address=145.79.227.49:4338 , piece_id=ANJJO7K52EL5XHCJQ5YXWN3PF6XJCJLITZ7HTW4IWJ7UFK4RIYWQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 29 09:13:48.056255992 (569h44m47.927339027s) [noise=true , address=145.79.227.49:4338 , piece_id=ANJJO7K52EL5XHCJQ5YXWN3PF6XJCJLITZ7HTW4IWJ7UFK4RIYWQ, seg_pos={0 0}, upload=499505]: Long tail cancellation due to optimal threshold Dec 29 09:13:48.056423150 (569h44m47.927506205s) [noise=true , address=145.79.227.44:4193 , piece_id=6Q2X63UQMN4LZZEJNR5DBRZTDEZNZQMCJZR5P6LNLXAK75JRZF6A, seg_pos={0 0}, upload=499505]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 29 09:13:48.056434101 (569h44m47.927517146s) [noise=true , address=145.79.227.44:4193 , piece_id=6Q2X63UQMN4LZZEJNR5DBRZTDEZNZQMCJZR5P6LNLXAK75JRZF6A, seg_pos={0 0}, upload=499505]: Long tail cancellation due to optimal threshold Dec 29 09:13:48.056583095 (569h44m47.92766613s) [noise=true , address=145.79.227.30:4762 , piece_id=5GQXZ4OB4NWJFRTCKWQCVNWXOZ2KIEAD56RHWBDC2ZLLAWDAV7AQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 09:13:48.056590429 (569h44m47.927673464s) [noise=true , address=145.79.227.30:4762 , piece_id=5GQXZ4OB4NWJFRTCKWQCVNWXOZ2KIEAD56RHWBDC2ZLLAWDAV7AQ, seg_pos={0 0}, upload=499505]: Long tail cancellation due to optimal threshold Dec 29 09:13:48.056608052 (569h44m47.927691087s) [seg_pos={0 0}, upload=499505]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:13:48.056673857 (569h44m47.927756892s) [upload=499505]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:13:48.056679157 (569h44m47.927762192s) [upload=499505]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:13:48.354466731 (569h44m48.225549786s) [upload=499505]: Done uploading segments... Dec 29 09:13:48.354490666 (569h44m48.225573711s) [upload=499505]: Upload finished. err: <nil> Duration: 5.931347699s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..XXX......X....X..XXX..X.....X..XX.X...X...XX.X.XX..XXXX..X] duration: 30.002277367s trace: https://jaeger.dev.storj.io/trace/36e37e344142e911 FAIL (2025-12-29 09:14:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: JFE6CW3QI26KBIXNS52XY6Y32DGYB2SMEFPLRL2MLSLWCU5WFGVA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: LMUZWTP5EAMIKHH7DVOWZQEXYUDKWRQ67BZ5QPC2MJF72YNYQ2AA) context deadline exceeded ======LOGS====== Dec 29 09:13:44.873699497 (569h44m44.744782532s) [upload=499505]: Starting upload Dec 29 09:13:44.873751726 (569h44m44.744834761s) [upload=499505]: Done starting upload Dec 29 09:13:44.873959902 (569h44m44.745042937s) [upload=499505]: Uploading segments... Dec 29 09:13:44.873998657 (569h44m44.745081692s) [upload=499505]: Got next segment. Inline: false Dec 29 09:13:44.874021700 (569h44m44.745104735s) [upload=499505]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:13:44.874026048 (569h44m44.745109084s) [upload=499505]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:13:45.118775289 (569h44m44.989858324s) [seg_pos={0 0}, upload=499505]: Begin upload segment... Dec 29 09:13:45.118821938 (569h44m44.989904973s) [seg_pos={0 0}, upload=499505]: Done begin upload segment. Dec 29 09:13:45.118895308 (569h44m44.989978353s) [noise=true , address=45.91.116.14:4190 , piece_id=JFE6CW3QI26KBIXNS52XY6Y32DGYB2SMEFPLRL2MLSLWCU5WFGVA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.119220547 (569h44m44.990303593s) [noise=true , address=145.79.227.49:4338 , piece_id=ANJJO7K52EL5XHCJQ5YXWN3PF6XJCJLITZ7HTW4IWJ7UFK4RIYWQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.119303475 (569h44m44.99038651s) [noise=true , address=145.79.227.45:4230 , piece_id=MM2BZQFTJATCGCVGVDLZJ4WHMPF6UAPLCFHA2NSOXMGDYETYCEWA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.119928295 (569h44m44.99101134s) [noise=true , address=145.79.227.52:4447 , piece_id=ZXP46EA4VMCENZUYJ6TYOGCWQ5ESJXGELM45IBEFWE7JNMHQUVLQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.120609763 (569h44m44.991692798s) [noise=true , address=45.91.116.9:4261 , piece_id=LMUZWTP5EAMIKHH7DVOWZQEXYUDKWRQ67BZ5QPC2MJF72YNYQ2AA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.121166202 (569h44m44.992249238s) [noise=true , address=145.79.227.51:4410 , piece_id=MAEXGPXSO3X2HCK3I6TZAHYKTGTU2OIWTHZAKCONF2RGO7SLAXHQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.121579408 (569h44m44.992662443s) [noise=true , address=145.79.227.41:4086 , piece_id=I6FIZXJZFPLOM4ZE3X4NAN4XKEFFW2HRQZXQAFSXTZJAVGMZ46NA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.121915829 (569h44m44.992998874s) [noise=true , address=145.79.227.44:4193 , piece_id=6Q2X63UQMN4LZZEJNR5DBRZTDEZNZQMCJZR5P6LNLXAK75JRZF6A, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.122221782 (569h44m44.993304817s) [noise=true , address=145.79.227.25:4619 , piece_id=U3FDPWHYIWOVEEPFZSGNQSJS6PEU5SHF5EEJ557SWYKVZA5HF4QQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.122347471 (569h44m44.993430506s) [noise=true , address=45.91.116.7:4010 , piece_id=47EEPQ6DYJXEDC4SBH754GXANR3SB5TPJM2V5UURQB4W2U55N6QA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.122631302 (569h44m44.993714337s) [noise=true , address=145.79.227.15:4876 , piece_id=JU44HVQTRIFQ3AN4A2CEUSB2IMSKX22QKLMQQG7PGINA7TXCPFHA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.122659946 (569h44m44.993742981s) [noise=true , address=145.79.227.6:4258 , piece_id=UXUDQ7LHMWDX3ZL5ZE5AZYBNLPZOCXDGBNPB2YOTXVXBEMKKEYBQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.123160349 (569h44m44.994243394s) [noise=true , address=77.90.31.12:4589 , piece_id=67S2V6GRLTPKRIQ43UJOLW7WJKFSU36JZK3BTUGLE6CUSQS7GCSA, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.123848570 (569h44m44.994931625s) [noise=true , address=145.79.227.11:4731 , piece_id=Y4DBCQ6JLJ5J3QXVNSYJ7RHVQVHKMVBUCATRWJSVMF73N37FC2TQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.123910607 (569h44m44.994993642s) [noise=true , address=145.79.227.26:4655 , piece_id=I5RYYWQIPV5FRJVPMMS7AMKZI2JXMLZSAPXQEHK3O4BI2E6QE2GQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:45.124656218 (569h44m44.995739253s) [noise=true , address=145.79.227.30:4762 , piece_id=5GQXZ4OB4NWJFRTCKWQCVNWXOZ2KIEAD56RHWBDC2ZLLAWDAV7AQ, seg_pos={0 0}, upload=499505]: Uploading piece... Dec 29 09:13:46.835334850 (569h44m46.706417905s) [upload=499505]: Next returned nil segment Dec 29 09:13:46.835360229 (569h44m46.706443264s) [upload=499505]: Waiting for error group managing segments... Dec 29 09:13:47.109839540 (569h44m46.980922585s) [noise=true , address=145.79.227.15:4876 , piece_id=JU44HVQTRIFQ3AN4A2CEUSB2IMSKX22QKLMQQG7PGINA7TXCPFHA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.110411438 (569h44m46.981494483s) [noise=true , address=145.79.227.11:4731 , piece_id=Y4DBCQ6JLJ5J3QXVNSYJ7RHVQVHKMVBUCATRWJSVMF73N37FC2TQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.114183481 (569h44m46.985266526s) [noise=true , address=145.79.227.52:4447 , piece_id=ZXP46EA4VMCENZUYJ6TYOGCWQ5ESJXGELM45IBEFWE7JNMHQUVLQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.114435100 (569h44m46.985518136s) [noise=true , address=145.79.227.45:4230 , piece_id=MM2BZQFTJATCGCVGVDLZJ4WHMPF6UAPLCFHA2NSOXMGDYETYCEWA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.122670862 (569h44m46.993753907s) [noise=true , address=145.79.227.41:4086 , piece_id=I6FIZXJZFPLOM4ZE3X4NAN4XKEFFW2HRQZXQAFSXTZJAVGMZ46NA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.129831077 (569h44m47.000914132s) [noise=true , address=145.79.227.25:4619 , piece_id=U3FDPWHYIWOVEEPFZSGNQSJS6PEU5SHF5EEJ557SWYKVZA5HF4QQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.131201766 (569h44m47.002284812s) [noise=true , address=145.79.227.6:4258 , piece_id=UXUDQ7LHMWDX3ZL5ZE5AZYBNLPZOCXDGBNPB2YOTXVXBEMKKEYBQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.132685763 (569h44m47.003768808s) [noise=true , address=145.79.227.51:4410 , piece_id=MAEXGPXSO3X2HCK3I6TZAHYKTGTU2OIWTHZAKCONF2RGO7SLAXHQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.134482052 (569h44m47.005565087s) [noise=true , address=145.79.227.26:4655 , piece_id=I5RYYWQIPV5FRJVPMMS7AMKZI2JXMLZSAPXQEHK3O4BI2E6QE2GQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.472679973 (569h44m47.343763018s) [noise=true , address=45.91.116.7:4010 , piece_id=47EEPQ6DYJXEDC4SBH754GXANR3SB5TPJM2V5UURQB4W2U55N6QA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.509758196 (569h44m47.380841231s) [noise=true , address=45.91.116.14:4190 , piece_id=JFE6CW3QI26KBIXNS52XY6Y32DGYB2SMEFPLRL2MLSLWCU5WFGVA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:47.957149808 (569h44m47.828232843s) [noise=true , address=77.90.31.12:4589 , piece_id=67S2V6GRLTPKRIQ43UJOLW7WJKFSU36JZK3BTUGLE6CUSQS7GCSA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:48.056078764 (569h44m47.927161799s) [noise=true , address=45.91.116.9:4261 , piece_id=LMUZWTP5EAMIKHH7DVOWZQEXYUDKWRQ67BZ5QPC2MJF72YNYQ2AA, seg_pos={0 0}, upload=499505]: Done uploading piece. err: <nil> Dec 29 09:13:48.056097490 (569h44m47.927180525s) [seg_pos={0 0}, upload=499505]: Segment reached optimal threshold of 13 pieces. Dec 29 09:13:48.056250281 (569h44m47.927333316s) [noise=true , address=145.79.227.49:4338 , piece_id=ANJJO7K52EL5XHCJQ5YXWN3PF6XJCJLITZ7HTW4IWJ7UFK4RIYWQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 29 09:13:48.056255992 (569h44m47.927339027s) [noise=true , address=145.79.227.49:4338 , piece_id=ANJJO7K52EL5XHCJQ5YXWN3PF6XJCJLITZ7HTW4IWJ7UFK4RIYWQ, seg_pos={0 0}, upload=499505]: Long tail cancellation due to optimal threshold Dec 29 09:13:48.056423150 (569h44m47.927506205s) [noise=true , address=145.79.227.44:4193 , piece_id=6Q2X63UQMN4LZZEJNR5DBRZTDEZNZQMCJZR5P6LNLXAK75JRZF6A, seg_pos={0 0}, upload=499505]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 29 09:13:48.056434101 (569h44m47.927517146s) [noise=true , address=145.79.227.44:4193 , piece_id=6Q2X63UQMN4LZZEJNR5DBRZTDEZNZQMCJZR5P6LNLXAK75JRZF6A, seg_pos={0 0}, upload=499505]: Long tail cancellation due to optimal threshold Dec 29 09:13:48.056583095 (569h44m47.92766613s) [noise=true , address=145.79.227.30:4762 , piece_id=5GQXZ4OB4NWJFRTCKWQCVNWXOZ2KIEAD56RHWBDC2ZLLAWDAV7AQ, seg_pos={0 0}, upload=499505]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 09:13:48.056590429 (569h44m47.927673464s) [noise=true , address=145.79.227.30:4762 , piece_id=5GQXZ4OB4NWJFRTCKWQCVNWXOZ2KIEAD56RHWBDC2ZLLAWDAV7AQ, seg_pos={0 0}, upload=499505]: Long tail cancellation due to optimal threshold Dec 29 09:13:48.056608052 (569h44m47.927691087s) [seg_pos={0 0}, upload=499505]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:13:48.056673857 (569h44m47.927756892s) [upload=499505]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:13:48.056679157 (569h44m47.927762192s) [upload=499505]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:13:48.354466731 (569h44m48.225549786s) [upload=499505]: Done uploading segments... Dec 29 09:13:48.354490666 (569h44m48.225573711s) [upload=499505]: Upload finished. err: <nil> Duration: 30.002300381s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 39.566325576s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 09:14:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 39.566357287s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [X............X.......................XX....X.............X..] duration: 3.351998923s trace: https://jaeger.dev.storj.io/trace/35fc14f889b57deb FAIL (2025-12-29 09:13:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.352024742s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [...XXX......X....X..XXX..X.....X..XX.X...X...XX.X.XX..XXXX..] duration: 6.058414259s trace: https://jaeger.dev.storj.io/trace/5ba0a367a5195416 FAIL (2025-12-29 09:13:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: P6ICJSJQ7MX4BKOJ2UQJ5GZJ4AX6SO23YSVBJQLM3FRCTLBEESLA) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: KS3WVMPDGVIOQSIP33BFMJYO5HDADZAN4GJO3ZOMLFHWHAT7ZOCA) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: 5JAUHSNZQOQB27CSH2WBUU3YLKY25T4FR2TPX6G5VQOC77TSYEZQ) context deadline exceeded ======LOGS====== Dec 29 09:11:53.498640416 (569h42m53.369723451s) [upload=499478]: Starting upload Dec 29 09:11:53.498691903 (569h42m53.369774948s) [upload=499478]: Done starting upload Dec 29 09:11:53.498832651 (569h42m53.369915686s) [upload=499478]: Uploading segments... Dec 29 09:11:53.498978970 (569h42m53.370062015s) [upload=499478]: Got next segment. Inline: false Dec 29 09:11:53.499017683 (569h42m53.370100718s) [upload=499478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:11:53.499022362 (569h42m53.370105407s) [upload=499478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:11:53.743418771 (569h42m53.614501806s) [seg_pos={0 0}, upload=499478]: Begin upload segment... Dec 29 09:11:53.743457074 (569h42m53.614540109s) [seg_pos={0 0}, upload=499478]: Done begin upload segment. Dec 29 09:11:53.743485518 (569h42m53.614568563s) [noise=true , address=45.91.116.10:4081 , piece_id=IKOYYGNRXB73R4IJ74QZELU7XZECFEA6BKBNNO4UVS2EEXL3RZ6A, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.743493473 (569h42m53.614576508s) [noise=true , address=145.79.227.19:4078 , piece_id=VLY32ZEJZX37GWYH7INDPGPQSAUYJTWMEJ2UPZERU7QZHA3HGTJA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.743912211 (569h42m53.614995246s) [noise=true , address=145.79.227.2:4115 , piece_id=UWGINS2BSGEHAVOIRU7EDIXERG2JLSQK3QUSVFNCA5S75D5LM2IA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.744224054 (569h42m53.61530709s) [noise=true , address=145.79.227.16:4911 , piece_id=PTCFGWQAZ2EIM364WKRKKJCZ7XTSDB4WKYOS2ZVYZAOWUYWJPNFA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.744606613 (569h42m53.615689648s) [noise=true , address=145.79.227.11:4729 , piece_id=UQNXWBXOIFHMDK2WXGFJ22IXV4B2LACKUPF5K34744PSPWTJKR6Q, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.744919569 (569h42m53.616002604s) [noise=true , address=145.79.227.12:4765 , piece_id=B2NQ44GLFOYVSS3DHGD3W43CMI5TJURFJ4MUYIFPK7IMDDE5ZSBA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.745234909 (569h42m53.616317944s) [noise=true , address=77.90.31.13:4629 , piece_id=P6ICJSJQ7MX4BKOJ2UQJ5GZJ4AX6SO23YSVBJQLM3FRCTLBEESLA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.745528217 (569h42m53.616611262s) [noise=true , address=45.91.116.8:4048 , piece_id=JIXTTITLNMLH5BV5RII2H4Y35QV2LZK5OG5IWJ75ABAHOW6Q3FVA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.745890949 (569h42m53.616973984s) [noise=true , address=45.91.116.9:4262 , piece_id=KS3WVMPDGVIOQSIP33BFMJYO5HDADZAN4GJO3ZOMLFHWHAT7ZOCA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.746126718 (569h42m53.617209753s) [noise=true , address=145.79.227.25:4619 , piece_id=2CCT5HIUB7YWRIFQVZIMKV2ZAX63HG2T55WMFFBPG7F2WSZMSHKQ, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.746452137 (569h42m53.617535172s) [noise=true , address=77.90.31.10:4517 , piece_id=5JAUHSNZQOQB27CSH2WBUU3YLKY25T4FR2TPX6G5VQOC77TSYEZQ, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.746763680 (569h42m53.617846715s) [noise=true , address=145.79.227.23:4547 , piece_id=OUBNAWQQXKIHNNVZOMM5SSIJYSNSVDXQZOKQPCXNA2TPQBJI3F4A, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.747136520 (569h42m53.618219555s) [noise=true , address=145.79.227.24:4582 , piece_id=QUK5ZZNGDORAEZD7TGVCTD3MCGKGRUF4SU2EB23FL7GYXQ3S3REA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.747452752 (569h42m53.618535787s) [noise=true , address=145.79.227.26:4654 , piece_id=XH56POZSV3DW4JUTGSALSQE4VUFZ4PDLY4NC3SP6VO4XO3DXSUSA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.747756691 (569h42m53.618839726s) [noise=true , address=145.79.227.39:4014 , piece_id=HZP6ZJB6MKJZQJXENDR4STCDFVCUXIEYHGDRTP4YSKMOWHDOVGVA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.749979594 (569h42m53.621062639s) [noise=true , address=145.79.227.40:4051 , piece_id=FFZWA4OT2Z4VDSCQMTLXFLT37CDKLTH2SABF3E4SV65FCP5GCE3A, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:55.352751696 (569h42m55.223834731s) [upload=499478]: Next returned nil segment Dec 29 09:11:55.352781292 (569h42m55.223864337s) [upload=499478]: Waiting for error group managing segments... Dec 29 09:11:55.732767744 (569h42m55.603850789s) [noise=true , address=145.79.227.11:4729 , piece_id=UQNXWBXOIFHMDK2WXGFJ22IXV4B2LACKUPF5K34744PSPWTJKR6Q, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.734799563 (569h42m55.605882608s) [noise=true , address=145.79.227.26:4654 , piece_id=XH56POZSV3DW4JUTGSALSQE4VUFZ4PDLY4NC3SP6VO4XO3DXSUSA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.735421037 (569h42m55.606504072s) [noise=true , address=145.79.227.23:4547 , piece_id=OUBNAWQQXKIHNNVZOMM5SSIJYSNSVDXQZOKQPCXNA2TPQBJI3F4A, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.736386064 (569h42m55.607469109s) [noise=true , address=145.79.227.19:4078 , piece_id=VLY32ZEJZX37GWYH7INDPGPQSAUYJTWMEJ2UPZERU7QZHA3HGTJA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.738456456 (569h42m55.609539492s) [noise=true , address=145.79.227.24:4582 , piece_id=QUK5ZZNGDORAEZD7TGVCTD3MCGKGRUF4SU2EB23FL7GYXQ3S3REA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.752674610 (569h42m55.623757645s) [noise=true , address=145.79.227.25:4619 , piece_id=2CCT5HIUB7YWRIFQVZIMKV2ZAX63HG2T55WMFFBPG7F2WSZMSHKQ, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.758728367 (569h42m55.629811402s) [noise=true , address=145.79.227.16:4911 , piece_id=PTCFGWQAZ2EIM364WKRKKJCZ7XTSDB4WKYOS2ZVYZAOWUYWJPNFA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.759947308 (569h42m55.631030343s) [noise=true , address=145.79.227.12:4765 , piece_id=B2NQ44GLFOYVSS3DHGD3W43CMI5TJURFJ4MUYIFPK7IMDDE5ZSBA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.764721469 (569h42m55.635804504s) [noise=true , address=145.79.227.39:4014 , piece_id=HZP6ZJB6MKJZQJXENDR4STCDFVCUXIEYHGDRTP4YSKMOWHDOVGVA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.770043333 (569h42m55.641126378s) [noise=true , address=145.79.227.2:4115 , piece_id=UWGINS2BSGEHAVOIRU7EDIXERG2JLSQK3QUSVFNCA5S75D5LM2IA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.990984339 (569h42m55.862067374s) [noise=true , address=77.90.31.13:4629 , piece_id=P6ICJSJQ7MX4BKOJ2UQJ5GZJ4AX6SO23YSVBJQLM3FRCTLBEESLA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.995189456 (569h42m55.866272501s) [noise=true , address=77.90.31.10:4517 , piece_id=5JAUHSNZQOQB27CSH2WBUU3YLKY25T4FR2TPX6G5VQOC77TSYEZQ, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:56.032501454 (569h42m55.903584489s) [noise=true , address=45.91.116.9:4262 , piece_id=KS3WVMPDGVIOQSIP33BFMJYO5HDADZAN4GJO3ZOMLFHWHAT7ZOCA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:56.032521722 (569h42m55.903604757s) [seg_pos={0 0}, upload=499478]: Segment reached optimal threshold of 13 pieces. Dec 29 09:11:56.032737903 (569h42m55.903820938s) [noise=true , address=45.91.116.8:4048 , piece_id=JIXTTITLNMLH5BV5RII2H4Y35QV2LZK5OG5IWJ75ABAHOW6Q3FVA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 29 09:11:56.032744877 (569h42m55.903827913s) [noise=true , address=45.91.116.8:4048 , piece_id=JIXTTITLNMLH5BV5RII2H4Y35QV2LZK5OG5IWJ75ABAHOW6Q3FVA, seg_pos={0 0}, upload=499478]: Long tail cancellation due to optimal threshold Dec 29 09:11:56.032931192 (569h42m55.904014227s) [noise=true , address=45.91.116.10:4081 , piece_id=IKOYYGNRXB73R4IJ74QZELU7XZECFEA6BKBNNO4UVS2EEXL3RZ6A, seg_pos={0 0}, upload=499478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 29 09:11:56.032947062 (569h42m55.904030107s) [noise=true , address=45.91.116.10:4081 , piece_id=IKOYYGNRXB73R4IJ74QZELU7XZECFEA6BKBNNO4UVS2EEXL3RZ6A, seg_pos={0 0}, upload=499478]: Long tail cancellation due to optimal threshold Dec 29 09:11:56.032960498 (569h42m55.904043543s) [noise=true , address=145.79.227.40:4051 , piece_id=FFZWA4OT2Z4VDSCQMTLXFLT37CDKLTH2SABF3E4SV65FCP5GCE3A, seg_pos={0 0}, upload=499478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 29 09:11:56.032967311 (569h42m55.904050346s) [noise=true , address=145.79.227.40:4051 , piece_id=FFZWA4OT2Z4VDSCQMTLXFLT37CDKLTH2SABF3E4SV65FCP5GCE3A, seg_pos={0 0}, upload=499478]: Long tail cancellation due to optimal threshold Dec 29 09:11:56.032993731 (569h42m55.904076766s) [seg_pos={0 0}, upload=499478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:11:56.033069595 (569h42m55.904152631s) [upload=499478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:11:56.033076549 (569h42m55.904159584s) [upload=499478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:11:56.325293174 (569h42m56.196376209s) [upload=499478]: Done uploading segments... Dec 29 09:11:56.325314154 (569h42m56.196397189s) [upload=499478]: Upload finished. err: <nil> Duration: 6.058443154s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 33.429311366s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 09:13:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 33.429340641s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.X............X.......................XX....X.............X.] duration: 3.461280238s trace: https://jaeger.dev.storj.io/trace/35fc14f889b57deb FAIL (2025-12-29 09:12:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.461297611s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [....XXX......X....X..XXX..X.....X..XX.X...X...XX.X.XX..XXXX.] duration: 5.720575591s trace: https://jaeger.dev.storj.io/trace/5ba0a367a5195416 FAIL (2025-12-29 09:12:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: P6ICJSJQ7MX4BKOJ2UQJ5GZJ4AX6SO23YSVBJQLM3FRCTLBEESLA) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: KS3WVMPDGVIOQSIP33BFMJYO5HDADZAN4GJO3ZOMLFHWHAT7ZOCA) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: 5JAUHSNZQOQB27CSH2WBUU3YLKY25T4FR2TPX6G5VQOC77TSYEZQ) context deadline exceeded ======LOGS====== Dec 29 09:11:53.498640416 (569h42m53.369723451s) [upload=499478]: Starting upload Dec 29 09:11:53.498691903 (569h42m53.369774948s) [upload=499478]: Done starting upload Dec 29 09:11:53.498832651 (569h42m53.369915686s) [upload=499478]: Uploading segments... Dec 29 09:11:53.498978970 (569h42m53.370062015s) [upload=499478]: Got next segment. Inline: false Dec 29 09:11:53.499017683 (569h42m53.370100718s) [upload=499478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:11:53.499022362 (569h42m53.370105407s) [upload=499478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:11:53.743418771 (569h42m53.614501806s) [seg_pos={0 0}, upload=499478]: Begin upload segment... Dec 29 09:11:53.743457074 (569h42m53.614540109s) [seg_pos={0 0}, upload=499478]: Done begin upload segment. Dec 29 09:11:53.743485518 (569h42m53.614568563s) [noise=true , address=45.91.116.10:4081 , piece_id=IKOYYGNRXB73R4IJ74QZELU7XZECFEA6BKBNNO4UVS2EEXL3RZ6A, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.743493473 (569h42m53.614576508s) [noise=true , address=145.79.227.19:4078 , piece_id=VLY32ZEJZX37GWYH7INDPGPQSAUYJTWMEJ2UPZERU7QZHA3HGTJA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.743912211 (569h42m53.614995246s) [noise=true , address=145.79.227.2:4115 , piece_id=UWGINS2BSGEHAVOIRU7EDIXERG2JLSQK3QUSVFNCA5S75D5LM2IA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.744224054 (569h42m53.61530709s) [noise=true , address=145.79.227.16:4911 , piece_id=PTCFGWQAZ2EIM364WKRKKJCZ7XTSDB4WKYOS2ZVYZAOWUYWJPNFA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.744606613 (569h42m53.615689648s) [noise=true , address=145.79.227.11:4729 , piece_id=UQNXWBXOIFHMDK2WXGFJ22IXV4B2LACKUPF5K34744PSPWTJKR6Q, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.744919569 (569h42m53.616002604s) [noise=true , address=145.79.227.12:4765 , piece_id=B2NQ44GLFOYVSS3DHGD3W43CMI5TJURFJ4MUYIFPK7IMDDE5ZSBA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.745234909 (569h42m53.616317944s) [noise=true , address=77.90.31.13:4629 , piece_id=P6ICJSJQ7MX4BKOJ2UQJ5GZJ4AX6SO23YSVBJQLM3FRCTLBEESLA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.745528217 (569h42m53.616611262s) [noise=true , address=45.91.116.8:4048 , piece_id=JIXTTITLNMLH5BV5RII2H4Y35QV2LZK5OG5IWJ75ABAHOW6Q3FVA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.745890949 (569h42m53.616973984s) [noise=true , address=45.91.116.9:4262 , piece_id=KS3WVMPDGVIOQSIP33BFMJYO5HDADZAN4GJO3ZOMLFHWHAT7ZOCA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.746126718 (569h42m53.617209753s) [noise=true , address=145.79.227.25:4619 , piece_id=2CCT5HIUB7YWRIFQVZIMKV2ZAX63HG2T55WMFFBPG7F2WSZMSHKQ, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.746452137 (569h42m53.617535172s) [noise=true , address=77.90.31.10:4517 , piece_id=5JAUHSNZQOQB27CSH2WBUU3YLKY25T4FR2TPX6G5VQOC77TSYEZQ, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.746763680 (569h42m53.617846715s) [noise=true , address=145.79.227.23:4547 , piece_id=OUBNAWQQXKIHNNVZOMM5SSIJYSNSVDXQZOKQPCXNA2TPQBJI3F4A, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.747136520 (569h42m53.618219555s) [noise=true , address=145.79.227.24:4582 , piece_id=QUK5ZZNGDORAEZD7TGVCTD3MCGKGRUF4SU2EB23FL7GYXQ3S3REA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.747452752 (569h42m53.618535787s) [noise=true , address=145.79.227.26:4654 , piece_id=XH56POZSV3DW4JUTGSALSQE4VUFZ4PDLY4NC3SP6VO4XO3DXSUSA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.747756691 (569h42m53.618839726s) [noise=true , address=145.79.227.39:4014 , piece_id=HZP6ZJB6MKJZQJXENDR4STCDFVCUXIEYHGDRTP4YSKMOWHDOVGVA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.749979594 (569h42m53.621062639s) [noise=true , address=145.79.227.40:4051 , piece_id=FFZWA4OT2Z4VDSCQMTLXFLT37CDKLTH2SABF3E4SV65FCP5GCE3A, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:55.352751696 (569h42m55.223834731s) [upload=499478]: Next returned nil segment Dec 29 09:11:55.352781292 (569h42m55.223864337s) [upload=499478]: Waiting for error group managing segments... Dec 29 09:11:55.732767744 (569h42m55.603850789s) [noise=true , address=145.79.227.11:4729 , piece_id=UQNXWBXOIFHMDK2WXGFJ22IXV4B2LACKUPF5K34744PSPWTJKR6Q, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.734799563 (569h42m55.605882608s) [noise=true , address=145.79.227.26:4654 , piece_id=XH56POZSV3DW4JUTGSALSQE4VUFZ4PDLY4NC3SP6VO4XO3DXSUSA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.735421037 (569h42m55.606504072s) [noise=true , address=145.79.227.23:4547 , piece_id=OUBNAWQQXKIHNNVZOMM5SSIJYSNSVDXQZOKQPCXNA2TPQBJI3F4A, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.736386064 (569h42m55.607469109s) [noise=true , address=145.79.227.19:4078 , piece_id=VLY32ZEJZX37GWYH7INDPGPQSAUYJTWMEJ2UPZERU7QZHA3HGTJA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.738456456 (569h42m55.609539492s) [noise=true , address=145.79.227.24:4582 , piece_id=QUK5ZZNGDORAEZD7TGVCTD3MCGKGRUF4SU2EB23FL7GYXQ3S3REA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.752674610 (569h42m55.623757645s) [noise=true , address=145.79.227.25:4619 , piece_id=2CCT5HIUB7YWRIFQVZIMKV2ZAX63HG2T55WMFFBPG7F2WSZMSHKQ, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.758728367 (569h42m55.629811402s) [noise=true , address=145.79.227.16:4911 , piece_id=PTCFGWQAZ2EIM364WKRKKJCZ7XTSDB4WKYOS2ZVYZAOWUYWJPNFA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.759947308 (569h42m55.631030343s) [noise=true , address=145.79.227.12:4765 , piece_id=B2NQ44GLFOYVSS3DHGD3W43CMI5TJURFJ4MUYIFPK7IMDDE5ZSBA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.764721469 (569h42m55.635804504s) [noise=true , address=145.79.227.39:4014 , piece_id=HZP6ZJB6MKJZQJXENDR4STCDFVCUXIEYHGDRTP4YSKMOWHDOVGVA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.770043333 (569h42m55.641126378s) [noise=true , address=145.79.227.2:4115 , piece_id=UWGINS2BSGEHAVOIRU7EDIXERG2JLSQK3QUSVFNCA5S75D5LM2IA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.990984339 (569h42m55.862067374s) [noise=true , address=77.90.31.13:4629 , piece_id=P6ICJSJQ7MX4BKOJ2UQJ5GZJ4AX6SO23YSVBJQLM3FRCTLBEESLA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.995189456 (569h42m55.866272501s) [noise=true , address=77.90.31.10:4517 , piece_id=5JAUHSNZQOQB27CSH2WBUU3YLKY25T4FR2TPX6G5VQOC77TSYEZQ, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:56.032501454 (569h42m55.903584489s) [noise=true , address=45.91.116.9:4262 , piece_id=KS3WVMPDGVIOQSIP33BFMJYO5HDADZAN4GJO3ZOMLFHWHAT7ZOCA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:56.032521722 (569h42m55.903604757s) [seg_pos={0 0}, upload=499478]: Segment reached optimal threshold of 13 pieces. Dec 29 09:11:56.032737903 (569h42m55.903820938s) [noise=true , address=45.91.116.8:4048 , piece_id=JIXTTITLNMLH5BV5RII2H4Y35QV2LZK5OG5IWJ75ABAHOW6Q3FVA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 29 09:11:56.032744877 (569h42m55.903827913s) [noise=true , address=45.91.116.8:4048 , piece_id=JIXTTITLNMLH5BV5RII2H4Y35QV2LZK5OG5IWJ75ABAHOW6Q3FVA, seg_pos={0 0}, upload=499478]: Long tail cancellation due to optimal threshold Dec 29 09:11:56.032931192 (569h42m55.904014227s) [noise=true , address=45.91.116.10:4081 , piece_id=IKOYYGNRXB73R4IJ74QZELU7XZECFEA6BKBNNO4UVS2EEXL3RZ6A, seg_pos={0 0}, upload=499478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 29 09:11:56.032947062 (569h42m55.904030107s) [noise=true , address=45.91.116.10:4081 , piece_id=IKOYYGNRXB73R4IJ74QZELU7XZECFEA6BKBNNO4UVS2EEXL3RZ6A, seg_pos={0 0}, upload=499478]: Long tail cancellation due to optimal threshold Dec 29 09:11:56.032960498 (569h42m55.904043543s) [noise=true , address=145.79.227.40:4051 , piece_id=FFZWA4OT2Z4VDSCQMTLXFLT37CDKLTH2SABF3E4SV65FCP5GCE3A, seg_pos={0 0}, upload=499478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 29 09:11:56.032967311 (569h42m55.904050346s) [noise=true , address=145.79.227.40:4051 , piece_id=FFZWA4OT2Z4VDSCQMTLXFLT37CDKLTH2SABF3E4SV65FCP5GCE3A, seg_pos={0 0}, upload=499478]: Long tail cancellation due to optimal threshold Dec 29 09:11:56.032993731 (569h42m55.904076766s) [seg_pos={0 0}, upload=499478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:11:56.033069595 (569h42m55.904152631s) [upload=499478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:11:56.033076549 (569h42m55.904159584s) [upload=499478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:11:56.325293174 (569h42m56.196376209s) [upload=499478]: Done uploading segments... Dec 29 09:11:56.325314154 (569h42m56.196397189s) [upload=499478]: Upload finished. err: <nil> Duration: 5.720601993s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..X............X.......................XX....X.............X] duration: 5.002274685s trace: https://jaeger.dev.storj.io/trace/35fc14f889b57deb FAIL (2025-12-29 09:12:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002288831s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.....XXX......X....X..XXX..X.....X..XX.X...X...XX.X.XX..XXXX] duration: 30.00257958s trace: https://jaeger.dev.storj.io/trace/5ba0a367a5195416 FAIL (2025-12-29 09:12:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: P6ICJSJQ7MX4BKOJ2UQJ5GZJ4AX6SO23YSVBJQLM3FRCTLBEESLA) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: KS3WVMPDGVIOQSIP33BFMJYO5HDADZAN4GJO3ZOMLFHWHAT7ZOCA) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: 5JAUHSNZQOQB27CSH2WBUU3YLKY25T4FR2TPX6G5VQOC77TSYEZQ) context deadline exceeded ======LOGS====== Dec 29 09:11:53.498640416 (569h42m53.369723451s) [upload=499478]: Starting upload Dec 29 09:11:53.498691903 (569h42m53.369774948s) [upload=499478]: Done starting upload Dec 29 09:11:53.498832651 (569h42m53.369915686s) [upload=499478]: Uploading segments... Dec 29 09:11:53.498978970 (569h42m53.370062015s) [upload=499478]: Got next segment. Inline: false Dec 29 09:11:53.499017683 (569h42m53.370100718s) [upload=499478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:11:53.499022362 (569h42m53.370105407s) [upload=499478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:11:53.743418771 (569h42m53.614501806s) [seg_pos={0 0}, upload=499478]: Begin upload segment... Dec 29 09:11:53.743457074 (569h42m53.614540109s) [seg_pos={0 0}, upload=499478]: Done begin upload segment. Dec 29 09:11:53.743485518 (569h42m53.614568563s) [noise=true , address=45.91.116.10:4081 , piece_id=IKOYYGNRXB73R4IJ74QZELU7XZECFEA6BKBNNO4UVS2EEXL3RZ6A, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.743493473 (569h42m53.614576508s) [noise=true , address=145.79.227.19:4078 , piece_id=VLY32ZEJZX37GWYH7INDPGPQSAUYJTWMEJ2UPZERU7QZHA3HGTJA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.743912211 (569h42m53.614995246s) [noise=true , address=145.79.227.2:4115 , piece_id=UWGINS2BSGEHAVOIRU7EDIXERG2JLSQK3QUSVFNCA5S75D5LM2IA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.744224054 (569h42m53.61530709s) [noise=true , address=145.79.227.16:4911 , piece_id=PTCFGWQAZ2EIM364WKRKKJCZ7XTSDB4WKYOS2ZVYZAOWUYWJPNFA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.744606613 (569h42m53.615689648s) [noise=true , address=145.79.227.11:4729 , piece_id=UQNXWBXOIFHMDK2WXGFJ22IXV4B2LACKUPF5K34744PSPWTJKR6Q, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.744919569 (569h42m53.616002604s) [noise=true , address=145.79.227.12:4765 , piece_id=B2NQ44GLFOYVSS3DHGD3W43CMI5TJURFJ4MUYIFPK7IMDDE5ZSBA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.745234909 (569h42m53.616317944s) [noise=true , address=77.90.31.13:4629 , piece_id=P6ICJSJQ7MX4BKOJ2UQJ5GZJ4AX6SO23YSVBJQLM3FRCTLBEESLA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.745528217 (569h42m53.616611262s) [noise=true , address=45.91.116.8:4048 , piece_id=JIXTTITLNMLH5BV5RII2H4Y35QV2LZK5OG5IWJ75ABAHOW6Q3FVA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.745890949 (569h42m53.616973984s) [noise=true , address=45.91.116.9:4262 , piece_id=KS3WVMPDGVIOQSIP33BFMJYO5HDADZAN4GJO3ZOMLFHWHAT7ZOCA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.746126718 (569h42m53.617209753s) [noise=true , address=145.79.227.25:4619 , piece_id=2CCT5HIUB7YWRIFQVZIMKV2ZAX63HG2T55WMFFBPG7F2WSZMSHKQ, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.746452137 (569h42m53.617535172s) [noise=true , address=77.90.31.10:4517 , piece_id=5JAUHSNZQOQB27CSH2WBUU3YLKY25T4FR2TPX6G5VQOC77TSYEZQ, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.746763680 (569h42m53.617846715s) [noise=true , address=145.79.227.23:4547 , piece_id=OUBNAWQQXKIHNNVZOMM5SSIJYSNSVDXQZOKQPCXNA2TPQBJI3F4A, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.747136520 (569h42m53.618219555s) [noise=true , address=145.79.227.24:4582 , piece_id=QUK5ZZNGDORAEZD7TGVCTD3MCGKGRUF4SU2EB23FL7GYXQ3S3REA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.747452752 (569h42m53.618535787s) [noise=true , address=145.79.227.26:4654 , piece_id=XH56POZSV3DW4JUTGSALSQE4VUFZ4PDLY4NC3SP6VO4XO3DXSUSA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.747756691 (569h42m53.618839726s) [noise=true , address=145.79.227.39:4014 , piece_id=HZP6ZJB6MKJZQJXENDR4STCDFVCUXIEYHGDRTP4YSKMOWHDOVGVA, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:53.749979594 (569h42m53.621062639s) [noise=true , address=145.79.227.40:4051 , piece_id=FFZWA4OT2Z4VDSCQMTLXFLT37CDKLTH2SABF3E4SV65FCP5GCE3A, seg_pos={0 0}, upload=499478]: Uploading piece... Dec 29 09:11:55.352751696 (569h42m55.223834731s) [upload=499478]: Next returned nil segment Dec 29 09:11:55.352781292 (569h42m55.223864337s) [upload=499478]: Waiting for error group managing segments... Dec 29 09:11:55.732767744 (569h42m55.603850789s) [noise=true , address=145.79.227.11:4729 , piece_id=UQNXWBXOIFHMDK2WXGFJ22IXV4B2LACKUPF5K34744PSPWTJKR6Q, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.734799563 (569h42m55.605882608s) [noise=true , address=145.79.227.26:4654 , piece_id=XH56POZSV3DW4JUTGSALSQE4VUFZ4PDLY4NC3SP6VO4XO3DXSUSA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.735421037 (569h42m55.606504072s) [noise=true , address=145.79.227.23:4547 , piece_id=OUBNAWQQXKIHNNVZOMM5SSIJYSNSVDXQZOKQPCXNA2TPQBJI3F4A, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.736386064 (569h42m55.607469109s) [noise=true , address=145.79.227.19:4078 , piece_id=VLY32ZEJZX37GWYH7INDPGPQSAUYJTWMEJ2UPZERU7QZHA3HGTJA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.738456456 (569h42m55.609539492s) [noise=true , address=145.79.227.24:4582 , piece_id=QUK5ZZNGDORAEZD7TGVCTD3MCGKGRUF4SU2EB23FL7GYXQ3S3REA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.752674610 (569h42m55.623757645s) [noise=true , address=145.79.227.25:4619 , piece_id=2CCT5HIUB7YWRIFQVZIMKV2ZAX63HG2T55WMFFBPG7F2WSZMSHKQ, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.758728367 (569h42m55.629811402s) [noise=true , address=145.79.227.16:4911 , piece_id=PTCFGWQAZ2EIM364WKRKKJCZ7XTSDB4WKYOS2ZVYZAOWUYWJPNFA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.759947308 (569h42m55.631030343s) [noise=true , address=145.79.227.12:4765 , piece_id=B2NQ44GLFOYVSS3DHGD3W43CMI5TJURFJ4MUYIFPK7IMDDE5ZSBA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.764721469 (569h42m55.635804504s) [noise=true , address=145.79.227.39:4014 , piece_id=HZP6ZJB6MKJZQJXENDR4STCDFVCUXIEYHGDRTP4YSKMOWHDOVGVA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.770043333 (569h42m55.641126378s) [noise=true , address=145.79.227.2:4115 , piece_id=UWGINS2BSGEHAVOIRU7EDIXERG2JLSQK3QUSVFNCA5S75D5LM2IA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.990984339 (569h42m55.862067374s) [noise=true , address=77.90.31.13:4629 , piece_id=P6ICJSJQ7MX4BKOJ2UQJ5GZJ4AX6SO23YSVBJQLM3FRCTLBEESLA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:55.995189456 (569h42m55.866272501s) [noise=true , address=77.90.31.10:4517 , piece_id=5JAUHSNZQOQB27CSH2WBUU3YLKY25T4FR2TPX6G5VQOC77TSYEZQ, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:56.032501454 (569h42m55.903584489s) [noise=true , address=45.91.116.9:4262 , piece_id=KS3WVMPDGVIOQSIP33BFMJYO5HDADZAN4GJO3ZOMLFHWHAT7ZOCA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: <nil> Dec 29 09:11:56.032521722 (569h42m55.903604757s) [seg_pos={0 0}, upload=499478]: Segment reached optimal threshold of 13 pieces. Dec 29 09:11:56.032737903 (569h42m55.903820938s) [noise=true , address=45.91.116.8:4048 , piece_id=JIXTTITLNMLH5BV5RII2H4Y35QV2LZK5OG5IWJ75ABAHOW6Q3FVA, seg_pos={0 0}, upload=499478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 29 09:11:56.032744877 (569h42m55.903827913s) [noise=true , address=45.91.116.8:4048 , piece_id=JIXTTITLNMLH5BV5RII2H4Y35QV2LZK5OG5IWJ75ABAHOW6Q3FVA, seg_pos={0 0}, upload=499478]: Long tail cancellation due to optimal threshold Dec 29 09:11:56.032931192 (569h42m55.904014227s) [noise=true , address=45.91.116.10:4081 , piece_id=IKOYYGNRXB73R4IJ74QZELU7XZECFEA6BKBNNO4UVS2EEXL3RZ6A, seg_pos={0 0}, upload=499478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 29 09:11:56.032947062 (569h42m55.904030107s) [noise=true , address=45.91.116.10:4081 , piece_id=IKOYYGNRXB73R4IJ74QZELU7XZECFEA6BKBNNO4UVS2EEXL3RZ6A, seg_pos={0 0}, upload=499478]: Long tail cancellation due to optimal threshold Dec 29 09:11:56.032960498 (569h42m55.904043543s) [noise=true , address=145.79.227.40:4051 , piece_id=FFZWA4OT2Z4VDSCQMTLXFLT37CDKLTH2SABF3E4SV65FCP5GCE3A, seg_pos={0 0}, upload=499478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 29 09:11:56.032967311 (569h42m55.904050346s) [noise=true , address=145.79.227.40:4051 , piece_id=FFZWA4OT2Z4VDSCQMTLXFLT37CDKLTH2SABF3E4SV65FCP5GCE3A, seg_pos={0 0}, upload=499478]: Long tail cancellation due to optimal threshold Dec 29 09:11:56.032993731 (569h42m55.904076766s) [seg_pos={0 0}, upload=499478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:11:56.033069595 (569h42m55.904152631s) [upload=499478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:11:56.033076549 (569h42m55.904159584s) [upload=499478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:11:56.325293174 (569h42m56.196376209s) [upload=499478]: Done uploading segments... Dec 29 09:11:56.325314154 (569h42m56.196397189s) [upload=499478]: Upload finished. err: <nil> Duration: 30.00259012s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 33.399932775s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 09:12:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 33.399965629s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.....XXX......X....X..XXX..X.....X..XX.X...X...XX.X.XX..XXX] duration: 30.0033832s trace: https://jaeger.dev.storj.io/trace/477aea535c9cb8ef FAIL (2025-12-29 09:11:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: AU2DMMETWAAN5MEMQBV34MTMNHHEJYWJCZF7CK5MTCECHVD4WBNQ) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: YBWOSORQR3YUQIN7CGRPXCEXSNMQB4ZZXAJFEUUTGJK3TLMWEF3Q) context deadline exceeded ======LOGS====== ec 29 09:11:23.247554002 (569h42m23.118637047s) [upload=499472]: Starting upload Dec 29 09:11:23.247622262 (569h42m23.118705317s) [upload=499472]: Done starting upload Dec 29 09:11:23.247726039 (569h42m23.118809074s) [upload=499472]: Uploading segments... Dec 29 09:11:23.247940317 (569h42m23.119023482s) [upload=499472]: Got next segment. Inline: false Dec 29 09:11:23.247987127 (569h42m23.119070162s) [upload=499472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:11:23.247991876 (569h42m23.119074911s) [upload=499472]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:11:23.499556122 (569h42m23.370639157s) [seg_pos={0 0}, upload=499472]: Begin upload segment... Dec 29 09:11:23.499632808 (569h42m23.370715843s) [seg_pos={0 0}, upload=499472]: Done begin upload segment. Dec 29 09:11:23.499691620 (569h42m23.370774655s) [noise=true , address=145.79.227.6:4257 , piece_id=KWYA5NUD27ZPKVHYDCQBIWVLHDBVLERRARPUAR7UH2V5XXEIA2KA, seg_pos={0 0}, upload=499472]: Uploading piece... Dec 29 09:11:23.499708331 (569h42m23.370791376s) [noise=true , address=77.90.31.14:4661 , piece_id=YBWOSORQR3YUQIN7CGRPXCEXSNMQB4ZZXAJFEUUTGJK3TLMWEF3Q, seg_pos={0 0}, upload=499472]: Uploading piece... Dec 29 09:11:23.500295089 (569h42m23.371378154s) [noise=true , address=145.79.227.51:4411 , piece_id=RLIULGF3DJBCFV2RZ6GYLYV6V3VNBUPYTP2WNYZMPHEFFEUKIEWA, seg_pos={0 0}, upload=499472]: Uploading piece... Dec 29 09:11:23.500838955 (569h42m23.37192199s) [noise=true , address=45.91.116.9:4264 , piece_id=HVEQIJHNEM5GITDI27FYVNNSFZ2WYUQBNTNK75XDOPZORQM23ZHQ, seg_pos={0 0}, upload=499472]: Uploading piece... Dec 29 09:11:23.501269715 (569h42m23.37235276s) [noise=true , address=145.79.227.33:4871 , piece_id=KDWYXM6EDUMMFIYPHMYAEZGQMVYPFIZ5LG3KGKIA5QUTAA2R42YA, seg_pos={0 0}, upload=499472]: Uploading piece... Dec 29 09:11:23.503498258 (569h42m23.374581293s) [noise=true , address=145.79.227.45:4229 , piece_id=CUD5UQQVPCO7Y4YFVTWXE7WAGSRVWJF4Z26GPCRXAITYINFCVDGQ, seg_pos={0 0}, upload=499472]: Uploading piece... Dec 29 09:11:23.503680456 (569h42m23.374763491s) [noise=true , address=77.90.31.18:4805 , piece_id=SG2K63PKMX6ILQU6OLDHOI2VHXQKHC4KTPFDGRT4FN3X52QBRH4A, seg_pos={0 0}, upload=499472]: Uploading piece... Dec 29 09:11:23.505049252 (569h42m23.376132287s) [noise=true , address=45.91.116.13:4155 , piece_id=FYBUK6DIEDZPAA2T45RY5PKAP5YKPKSR4O3AYT7NZV2CUBWEL7YA, seg_pos={0 0}, upload=499472]: Uploading piece... Dec 29 09:11:23.506716216 (569h42m23.377799261s) [noise=true , address=145.79.227.26:4655 , piece_id=O5XDFKHV7NOURGXB6LR57QKP26T7BOS4IANBIR2HLZTTDQL6YXMQ, seg_pos={0 0}, upload=499472]: Uploading piece... Dec 29 09:11:23.508491416 (569h42m23.379574461s) [noise=true , address=77.90.31.12:4589 , piece_id=JMCIWHOVESIIVIH5YDHK7LLP3RXY3T75X3RKMHIXRWYS4QAWKUAQ, seg_pos={0 0}, upload=499472]: Uploading piece... Dec 29 09:11:23.509306939 (569h42m23.380389974s) [noise=true , address=145.79.227.38:4977 , piece_id=7IJXBXVMS66FPML7AUTZBSSMFJ4EYFGASWYONBM6D3XG2PO5FF4Q, seg_pos={0 0}, upload=499472]: Uploading piece... Dec 29 09:11:23.510760316 (569h42m23.381843361s) [noise=true , address=145.79.227.14:4837 , piece_id=EBFBF7RKAH25TSIVPDLKPFPOIF5QT7C5OE4QBJ5YDMKYZMR6BM3Q, seg_pos={0 0}, upload=499472]: Uploading piece... Dec 29 09:11:23.511168624 (569h42m23.382251669s) [noise=true , address=145.79.227.7:4294 , piece_id=V3NUKWSIPA7SS3HNCLUIWEYRTPSCMKVN3FJUJTZFW4JSSRKCJQIQ, seg_pos={0 0}, upload=499472]: Uploading piece... Dec 29 09:11:23.511625103 (569h42m23.382708138s) [noise=true , address=45.91.116.7:4010 , piece_id=AU2DMMETWAAN5MEMQBV34MTMNHHEJYWJCZF7CK5MTCECHVD4WBNQ, seg_pos={0 0}, upload=499472]: Uploading piece... Dec 29 09:11:23.513794184 (569h42m23.384877229s) [noise=true , address=77.90.31.21:4877 , piece_id=AP2JEPKQDO42N7SEVPHXR7N4XNUF5KEDLZQNLP2U27BWWYCVBIGQ, seg_pos={0 0}, upload=499472]: Uploading piece... Dec 29 09:11:23.515032741 (569h42m23.386115787s) [noise=true , address=45.91.116.11:4119 , piece_id=34SO3MB3U2XH6IAN6C6ZUECXO2R6P6GBK7ZXEQ2XIVBOLKGSQCCA, seg_pos={0 0}, upload=499472]: Uploading piece... Dec 29 09:11:25.202653321 (569h42m25.073736366s) [upload=499472]: Next returned nil segment Dec 29 09:11:25.202674672 (569h42m25.073757717s) [upload=499472]: Waiting for error group managing segments... Dec 29 09:11:25.485564157 (569h42m25.356647192s) [noise=true , address=145.79.227.51:4411 , piece_id=RLIULGF3DJBCFV2RZ6GYLYV6V3VNBUPYTP2WNYZMPHEFFEUKIEWA, seg_pos={0 0}, upload=499472]: Done uploading piece. err: <nil> Dec 29 09:11:25.497258223 (569h42m25.368341268s) [noise=true , address=145.79.227.38:4977 , piece_id=7IJXBXVMS66FPML7AUTZBSSMFJ4EYFGASWYONBM6D3XG2PO5FF4Q, seg_pos={0 0}, upload=499472]: Done uploading piece. err: <nil> Dec 29 09:11:25.498208834 (569h42m25.369291869s) [noise=true , address=145.79.227.14:4837 , piece_id=EBFBF7RKAH25TSIVPDLKPFPOIF5QT7C5OE4QBJ5YDMKYZMR6BM3Q, seg_pos={0 0}, upload=499472]: Done uploading piece. err: <nil> Dec 29 09:11:25.502556653 (569h42m25.373639698s) [noise=true , address=145.79.227.45:4229 , piece_id=CUD5UQQVPCO7Y4YFVTWXE7WAGSRVWJF4Z26GPCRXAITYINFCVDGQ, seg_pos={0 0}, upload=499472]: Done uploading piece. err: <nil> Dec 29 09:11:25.512273605 (569h42m25.38335665s) [noise=true , address=145.79.227.6:4257 , piece_id=KWYA5NUD27ZPKVHYDCQBIWVLHDBVLERRARPUAR7UH2V5XXEIA2KA, seg_pos={0 0}, upload=499472]: Done uploading piece. err: <nil> Dec 29 09:11:25.515687004 (569h42m25.386770049s) [noise=true , address=145.79.227.26:4655 , piece_id=O5XDFKHV7NOURGXB6LR57QKP26T7BOS4IANBIR2HLZTTDQL6YXMQ, seg_pos={0 0}, upload=499472]: Done uploading piece. err: <nil> Dec 29 09:11:25.529590189 (569h42m25.400673224s) [noise=true , address=145.79.227.7:4294 , piece_id=V3NUKWSIPA7SS3HNCLUIWEYRTPSCMKVN3FJUJTZFW4JSSRKCJQIQ, seg_pos={0 0}, upload=499472]: Done uploading piece. err: <nil> Dec 29 09:11:25.782554401 (569h42m25.653637436s) [noise=true , address=77.90.31.21:4877 , piece_id=AP2JEPKQDO42N7SEVPHXR7N4XNUF5KEDLZQNLP2U27BWWYCVBIGQ, seg_pos={0 0}, upload=499472]: Done uploading piece. err: <nil> Dec 29 09:11:25.847679930 (569h42m25.718762965s) [noise=true , address=45.91.116.11:4119 , piece_id=34SO3MB3U2XH6IAN6C6ZUECXO2R6P6GBK7ZXEQ2XIVBOLKGSQCCA, seg_pos={0 0}, upload=499472]: Done uploading piece. err: <nil> Dec 29 09:11:25.865452372 (569h42m25.736535407s) [noise=true , address=45.91.116.9:4264 , piece_id=HVEQIJHNEM5GITDI27FYVNNSFZ2WYUQBNTNK75XDOPZORQM23ZHQ, seg_pos={0 0}, upload=499472]: Done uploading piece. err: <nil> Dec 29 09:11:25.966838396 (569h42m25.837921431s) [noise=true , address=45.91.116.7:4010 , piece_id=AU2DMMETWAAN5MEMQBV34MTMNHHEJYWJCZF7CK5MTCECHVD4WBNQ, seg_pos={0 0}, upload=499472]: Done uploading piece. err: <nil> Dec 29 09:11:25.988188580 (569h42m25.859271616s) [noise=true , address=77.90.31.14:4661 , piece_id=YBWOSORQR3YUQIN7CGRPXCEXSNMQB4ZZXAJFEUUTGJK3TLMWEF3Q, seg_pos={0 0}, upload=499472]: Done uploading piece. err: <nil> Dec 29 09:11:26.044959361 (569h42m25.916042406s) [noise=true , address=45.91.116.13:4155 , piece_id=FYBUK6DIEDZPAA2T45RY5PKAP5YKPKSR4O3AYT7NZV2CUBWEL7YA, seg_pos={0 0}, upload=499472]: Done uploading piece. err: <nil> Dec 29 09:11:26.044980371 (569h42m25.916063406s) [seg_pos={0 0}, upload=499472]: Segment reached optimal threshold of 13 pieces. Dec 29 09:11:26.045154884 (569h42m25.916237919s) [noise=true , address=77.90.31.12:4589 , piece_id=JMCIWHOVESIIVIH5YDHK7LLP3RXY3T75X3RKMHIXRWYS4QAWKUAQ, seg_pos={0 0}, upload=499472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 29 09:11:26.045162298 (569h42m25.916245333s) [noise=true , address=77.90.31.12:4589 , piece_id=JMCIWHOVESIIVIH5YDHK7LLP3RXY3T75X3RKMHIXRWYS4QAWKUAQ, seg_pos={0 0}, upload=499472]: Long tail cancellation due to optimal threshold Dec 29 09:11:26.045341429 (569h42m25.916424474s) [noise=true , address=145.79.227.33:4871 , piece_id=KDWYXM6EDUMMFIYPHMYAEZGQMVYPFIZ5LG3KGKIA5QUTAA2R42YA, seg_pos={0 0}, upload=499472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 29 09:11:26.045348111 (569h42m25.916431156s) [noise=true , address=145.79.227.33:4871 , piece_id=KDWYXM6EDUMMFIYPHMYAEZGQMVYPFIZ5LG3KGKIA5QUTAA2R42YA, seg_pos={0 0}, upload=499472]: Long tail cancellation due to optimal threshold Dec 29 09:11:26.045362549 (569h42m25.916445584s) [noise=true , address=77.90.31.18:4805 , piece_id=SG2K63PKMX6ILQU6OLDHOI2VHXQKHC4KTPFDGRT4FN3X52QBRH4A, seg_pos={0 0}, upload=499472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:34112->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:34112->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34112->77.90.31.18:4805: use of closed network connection Dec 29 09:11:26.045378098 (569h42m25.916461133s) [noise=true , address=77.90.31.18:4805 , piece_id=SG2K63PKMX6ILQU6OLDHOI2VHXQKHC4KTPFDGRT4FN3X52QBRH4A, seg_pos={0 0}, upload=499472]: Long tail cancellation due to optimal threshold Dec 29 09:11:26.045402425 (569h42m25.91648546s) [seg_pos={0 0}, upload=499472]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:11:26.045483078 (569h42m25.916566113s) [upload=499472]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:11:26.045487436 (569h42m25.916570471s) [upload=499472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:11:26.339755809 (569h42m26.210838854s) [upload=499472]: Done uploading segments... Dec 29 09:11:26.339785355 (569h42m26.2108684s) [upload=499472]: Upload finished. err: <nil> Duration: 30.003409361s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...X............X.......................XX....X.............] duration: 3.528833142s trace: https://jaeger.dev.storj.io/trace/2818f7531409d35c FAIL (2025-12-29 09:11:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.528860434s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [....X............X.......................XX....X............] duration: 3.433904611s trace: https://jaeger.dev.storj.io/trace/2818f7531409d35c FAIL (2025-12-29 09:11:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.433934758s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 29.798943037s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 09:11:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 29.798968777s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.....XXX......X....X..XXX..X.....X..XX.X...X...XX.X.XX..XX] duration: 30.003427589s trace: https://jaeger.dev.storj.io/trace/64e92cb3c47a188a FAIL (2025-12-29 09:10:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 09:10:27.106377441 (569h41m26.977460476s) [upload=499458]: Starting upload Dec 29 09:10:27.106431523 (569h41m26.977514568s) [upload=499458]: Done starting upload Dec 29 09:10:27.106501667 (569h41m26.977584702s) [upload=499458]: Uploading segments... Dec 29 09:10:27.106644018 (569h41m26.977727053s) [upload=499458]: Got next segment. Inline: false Dec 29 09:10:27.106674747 (569h41m26.977757782s) [upload=499458]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:10:27.106679916 (569h41m26.977762951s) [upload=499458]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:10:27.355293054 (569h41m27.226376089s) [seg_pos={0 0}, upload=499458]: Begin upload segment... Dec 29 09:10:27.355353269 (569h41m27.226436304s) [seg_pos={0 0}, upload=499458]: Done begin upload segment. Dec 29 09:10:27.355390019 (569h41m27.226473054s) [noise=true , address=145.79.227.41:4085 , piece_id=YWNOZWMXQ4V2F6KFRASPZ7XM3ZJDV2FRU2IGRA4OETZQSKZ5XOHA, seg_pos={0 0}, upload=499458]: Uploading piece... Dec 29 09:10:27.356353413 (569h41m27.227436458s) [noise=true , address=45.91.116.12:4228 , piece_id=GHKBY335BUNMKPZS66TCWRDYXCMRE7LXVYFW2ZZVGSGU3NIAWGXA, seg_pos={0 0}, upload=499458]: Uploading piece... Dec 29 09:10:27.356779915 (569h41m27.22786295s) [noise=true , address=45.91.116.9:4263 , piece_id=QIIJXFQRMKFRN3T2SZ62P4UNLXHBAQA2XQSHJ77WJZ5TRBMTBVLQ, seg_pos={0 0}, upload=499458]: Uploading piece... Dec 29 09:10:27.357274797 (569h41m27.228357832s) [noise=true , address=145.79.227.39:4015 , piece_id=JLWKEAXYIT7Z4NGIZA3WR2NWLMVYFFSTPNRO6FT5OWQWCLXUR5RA, seg_pos={0 0}, upload=499458]: Uploading piece... Dec 29 09:10:27.357733561 (569h41m27.228816596s) [noise=true , address=45.91.116.14:4192 , piece_id=QWOKHKKQPT3Z3XA346PEV6LV6RJC3RR47XVOJ4UCIZHQPB6O4K3A, seg_pos={0 0}, upload=499458]: Uploading piece... Dec 29 09:10:27.358265153 (569h41m27.229348188s) [noise=true , address=77.90.31.14:4661 , piece_id=56RWEKNUMVX3SUUFT7CSPLCB34HZYYSAXRAYLUWJQQHIK4RSVYIA, seg_pos={0 0}, upload=499458]: Uploading piece... Dec 29 09:10:27.359938921 (569h41m27.231021966s) [noise=true , address=45.91.116.10:4082 , piece_id=FOANWW2OR3ORFVYUZ36MRSDWCVYS4TKZPS62PV2B4XZGVE5FNEMQ, seg_pos={0 0}, upload=499458]: Uploading piece... Dec 29 09:10:27.361450028 (569h41m27.232533073s) [noise=true , address=45.91.116.13:4155 , piece_id=N7PR5TFXJMUMFQYQKVU7T53MFYYV5SFO5CHTNIDOWUDBIWDKQH6Q, seg_pos={0 0}, upload=499458]: Uploading piece... Dec 29 09:10:27.363404399 (569h41m27.234487444s) [noise=true , address=145.79.227.51:4409 , piece_id=M4NU7ILNVBK5SVGAZUVYOCNIVGQNZVFE3MVD2AEP72QZC6EGF3TA, seg_pos={0 0}, upload=499458]: Uploading piece... Dec 29 09:10:27.364335352 (569h41m27.235418387s) [noise=true , address=145.79.227.13:4802 , piece_id=TXLSVRDPQNTVJZSW2STIRR6BVB3OPSBI3GFXWSYZIJFMB5CNEPOA, seg_pos={0 0}, upload=499458]: Uploading piece... Dec 29 09:10:27.365713456 (569h41m27.236796491s) [noise=true , address=45.91.116.11:4117 , piece_id=Y43EAYY4WID6VVHHDDDSHINRSIFP75VSVGLYA6LFDCM5KDOFMMYA, seg_pos={0 0}, upload=499458]: Uploading piece... Dec 29 09:10:27.366639910 (569h41m27.237722955s) [noise=true , address=77.90.31.10:4517 , piece_id=GAI5FEFYFHLVBKKRPTC2EBPERXRUN4GRDCTBDYPB2M4DUCKOTR6A, seg_pos={0 0}, upload=499458]: Uploading piece... Dec 29 09:10:27.367728212 (569h41m27.238811247s) [noise=true , address=145.79.227.25:4619 , piece_id=2V45TMSWONYBBHEOHQYKCP6ZVXIZWUNW7GC5SKTBMYE7J6YM6MCQ, seg_pos={0 0}, upload=499458]: Uploading piece... Dec 29 09:10:27.368903260 (569h41m27.239986295s) [noise=true , address=45.91.116.5:4298 , piece_id=OUF7RMWAK7PK3HOLU4ODLX7UXY5GURLZGSHNLGMM5WB4IDCTHOUQ, seg_pos={0 0}, upload=499458]: Uploading piece... Dec 29 09:10:27.369633260 (569h41m27.240716315s) [noise=true , address=145.79.227.2:4113 , piece_id=PQJX5XT6KMMRXIOQT43I355OWPMBIJMP6PMLZS5FHFGQTWV2HSEQ, seg_pos={0 0}, upload=499458]: Uploading piece... Dec 29 09:10:27.370611683 (569h41m27.241694728s) [noise=true , address=145.79.227.35:4483 , piece_id=BORSG5UW6ITW7LMODJZHCRMJWYHYLIWT7KIC6TDBU5T5ML5TVOXA, seg_pos={0 0}, upload=499458]: Uploading piece... Dec 29 09:10:29.062798527 (569h41m28.933881572s) [upload=499458]: Next returned nil segment Dec 29 09:10:29.062812553 (569h41m28.933895598s) [upload=499458]: Waiting for error group managing segments... Dec 29 09:10:29.347746361 (569h41m29.218829416s) [noise=true , address=145.79.227.13:4802 , piece_id=TXLSVRDPQNTVJZSW2STIRR6BVB3OPSBI3GFXWSYZIJFMB5CNEPOA, seg_pos={0 0}, upload=499458]: Done uploading piece. err: <nil> Dec 29 09:10:29.350340471 (569h41m29.221423506s) [noise=true , address=145.79.227.25:4619 , piece_id=2V45TMSWONYBBHEOHQYKCP6ZVXIZWUNW7GC5SKTBMYE7J6YM6MCQ, seg_pos={0 0}, upload=499458]: Done uploading piece. err: <nil> Dec 29 09:10:29.362310163 (569h41m29.233393198s) [noise=true , address=145.79.227.2:4113 , piece_id=PQJX5XT6KMMRXIOQT43I355OWPMBIJMP6PMLZS5FHFGQTWV2HSEQ, seg_pos={0 0}, upload=499458]: Done uploading piece. err: <nil> Dec 29 09:10:29.368567839 (569h41m29.239650874s) [noise=true , address=145.79.227.51:4409 , piece_id=M4NU7ILNVBK5SVGAZUVYOCNIVGQNZVFE3MVD2AEP72QZC6EGF3TA, seg_pos={0 0}, upload=499458]: Done uploading piece. err: <nil> Dec 29 09:10:29.375322471 (569h41m29.246405506s) [noise=true , address=145.79.227.39:4015 , piece_id=JLWKEAXYIT7Z4NGIZA3WR2NWLMVYFFSTPNRO6FT5OWQWCLXUR5RA, seg_pos={0 0}, upload=499458]: Done uploading piece. err: <nil> Dec 29 09:10:29.382650744 (569h41m29.253733779s) [noise=true , address=145.79.227.41:4085 , piece_id=YWNOZWMXQ4V2F6KFRASPZ7XM3ZJDV2FRU2IGRA4OETZQSKZ5XOHA, seg_pos={0 0}, upload=499458]: Done uploading piece. err: <nil> Dec 29 09:10:29.543345281 (569h41m29.414428326s) [noise=true , address=77.90.31.10:4517 , piece_id=GAI5FEFYFHLVBKKRPTC2EBPERXRUN4GRDCTBDYPB2M4DUCKOTR6A, seg_pos={0 0}, upload=499458]: Done uploading piece. err: <nil> Dec 29 09:10:29.546423654 (569h41m29.417506689s) [noise=true , address=145.79.227.35:4483 , piece_id=BORSG5UW6ITW7LMODJZHCRMJWYHYLIWT7KIC6TDBU5T5ML5TVOXA, seg_pos={0 0}, upload=499458]: Done uploading piece. err: <nil> Dec 29 09:10:29.582678459 (569h41m29.453761504s) [noise=true , address=45.91.116.11:4117 , piece_id=Y43EAYY4WID6VVHHDDDSHINRSIFP75VSVGLYA6LFDCM5KDOFMMYA, seg_pos={0 0}, upload=499458]: Done uploading piece. err: <nil> Dec 29 09:10:29.594116428 (569h41m29.465199463s) [noise=true , address=77.90.31.14:4661 , piece_id=56RWEKNUMVX3SUUFT7CSPLCB34HZYYSAXRAYLUWJQQHIK4RSVYIA, seg_pos={0 0}, upload=499458]: Done uploading piece. err: <nil> Dec 29 09:10:29.607348975 (569h41m29.47843201s) [noise=true , address=45.91.116.9:4263 , piece_id=QIIJXFQRMKFRN3T2SZ62P4UNLXHBAQA2XQSHJ77WJZ5TRBMTBVLQ, seg_pos={0 0}, upload=499458]: Done uploading piece. err: <nil> Dec 29 09:10:29.738066150 (569h41m29.609149205s) [noise=true , address=45.91.116.13:4155 , piece_id=N7PR5TFXJMUMFQYQKVU7T53MFYYV5SFO5CHTNIDOWUDBIWDKQH6Q, seg_pos={0 0}, upload=499458]: Done uploading piece. err: <nil> Dec 29 09:10:29.745271490 (569h41m29.616354525s) [noise=true , address=45.91.116.5:4298 , piece_id=OUF7RMWAK7PK3HOLU4ODLX7UXY5GURLZGSHNLGMM5WB4IDCTHOUQ, seg_pos={0 0}, upload=499458]: Done uploading piece. err: <nil> Dec 29 09:10:29.745292661 (569h41m29.616375696s) [seg_pos={0 0}, upload=499458]: Segment reached optimal threshold of 13 pieces. Dec 29 09:10:29.745682152 (569h41m29.616765197s) [noise=true , address=45.91.116.10:4082 , piece_id=FOANWW2OR3ORFVYUZ36MRSDWCVYS4TKZPS62PV2B4XZGVE5FNEMQ, seg_pos={0 0}, upload=499458]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 29 09:10:29.745691109 (569h41m29.616774154s) [noise=true , address=45.91.116.10:4082 , piece_id=FOANWW2OR3ORFVYUZ36MRSDWCVYS4TKZPS62PV2B4XZGVE5FNEMQ, seg_pos={0 0}, upload=499458]: Long tail cancellation due to optimal threshold Dec 29 09:10:29.745877493 (569h41m29.616960539s) [noise=true , address=45.91.116.12:4228 , piece_id=GHKBY335BUNMKPZS66TCWRDYXCMRE7LXVYFW2ZZVGSGU3NIAWGXA, seg_pos={0 0}, upload=499458]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 29 09:10:29.745885409 (569h41m29.616968454s) [noise=true , address=45.91.116.12:4228 , piece_id=GHKBY335BUNMKPZS66TCWRDYXCMRE7LXVYFW2ZZVGSGU3NIAWGXA, seg_pos={0 0}, upload=499458]: Long tail cancellation due to optimal threshold Dec 29 09:10:29.746094878 (569h41m29.617177913s) [noise=true , address=45.91.116.14:4192 , piece_id=QWOKHKKQPT3Z3XA346PEV6LV6RJC3RR47XVOJ4UCIZHQPB6O4K3A, seg_pos={0 0}, upload=499458]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: read tcp 188.166.225.170:53882->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 09:10:29.746103023 (569h41m29.617186069s) [noise=true , address=45.91.116.14:4192 , piece_id=QWOKHKKQPT3Z3XA346PEV6LV6RJC3RR47XVOJ4UCIZHQPB6O4K3A, seg_pos={0 0}, upload=499458]: Long tail cancellation due to optimal threshold Dec 29 09:10:29.746117891 (569h41m29.617200926s) [seg_pos={0 0}, upload=499458]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:10:29.746205559 (569h41m29.617288604s) [upload=499458]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:10:29.746210738 (569h41m29.617293784s) [upload=499458]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:10:30.036282449 (569h41m29.907365484s) [upload=499458]: Done uploading segments... Dec 29 09:10:30.036301245 (569h41m29.90738428s) [upload=499458]: Upload finished. err: <nil> Duration: 30.003454529s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 36.457729226s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 09:10:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 36.45775227s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.....X............X.......................XX....X...........] duration: 3.400520784s trace: https://jaeger.dev.storj.io/trace/2818f7531409d35c FAIL (2025-12-29 09:10:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.400552002s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X.....XXX......X....X..XXX..X.....X..XX.X...X...XX.X.XX..X] duration: 30.002529288s trace: https://jaeger.dev.storj.io/trace/3f9c9f21a5b02a43 FAIL (2025-12-29 09:09:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: QS2DMK6RLXOQGERYYSZGKK6KQLSFUVAQHP7HII6YC7KROYJOBQ5Q) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: GACMF3J5TOBKH7DBFJUBLAOVM6NPANPHNSKCGVXYZF6NTX6XWSVA) context deadline exceeded ======LOGS====== g segments... Dec 29 09:09:23.953325319 (569h40m23.824408354s) [upload=499442]: Got next segment. Inline: false Dec 29 09:09:23.953353783 (569h40m23.824436828s) [upload=499442]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:09:23.953358252 (569h40m23.824441287s) [upload=499442]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:09:24.200240713 (569h40m24.071323759s) [seg_pos={0 0}, upload=499442]: Begin upload segment... Dec 29 09:09:24.200295408 (569h40m24.071378463s) [seg_pos={0 0}, upload=499442]: Done begin upload segment. Dec 29 09:09:24.200331477 (569h40m24.071414532s) [noise=true , address=77.90.31.14:4662 , piece_id=TTY7Z2MLHY5HWWU6OAQZC7PGJOJCQL2VNKDZZYDEH32D7V2RLWTQ, seg_pos={0 0}, upload=499442]: Uploading piece... Dec 29 09:09:24.200446485 (569h40m24.07152952s) [noise=true , address=45.91.116.12:4226 , piece_id=SZTDTDIAAVRUWK3E27PVYZGYCSC4R3POM2ODTVWWKNIB3T2W7QRQ, seg_pos={0 0}, upload=499442]: Uploading piece... Dec 29 09:09:24.200895320 (569h40m24.071978355s) [noise=true , address=145.79.227.13:4804 , piece_id=AAEETYPTHUGKBUKRAKZLTJ6TVDUG2FSGFNIAFOKUKDQKNZWZU4PQ, seg_pos={0 0}, upload=499442]: Uploading piece... Dec 29 09:09:24.201335819 (569h40m24.072418854s) [noise=true , address=145.79.227.45:4229 , piece_id=US6NQVN4YBOTBS7GG57BRI7FZ77OHU6WYFMA5BTQKWYHSQFXQNZA, seg_pos={0 0}, upload=499442]: Uploading piece... Dec 29 09:09:24.201702507 (569h40m24.072785542s) [noise=true , address=145.79.227.46:4603 , piece_id=4RZPDUGR5CI7IYOIWINGQINDWBEPWQLSWY2T7ZDZY262DV7C5UNQ, seg_pos={0 0}, upload=499442]: Uploading piece... Dec 29 09:09:24.202154428 (569h40m24.073237463s) [noise=true , address=45.91.116.6:4975 , piece_id=QS2DMK6RLXOQGERYYSZGKK6KQLSFUVAQHP7HII6YC7KROYJOBQ5Q, seg_pos={0 0}, upload=499442]: Uploading piece... Dec 29 09:09:24.202618972 (569h40m24.073702007s) [noise=true , address=145.79.227.16:4912 , piece_id=LAW3OSFHRDZ345SKYCFNDJO5MDG4GLAUJ243C54DIOZ34GK543BA, seg_pos={0 0}, upload=499442]: Uploading piece... Dec 29 09:09:24.203020146 (569h40m24.074103181s) [noise=true , address=45.91.116.10:4081 , piece_id=PRIHC33CW2ZMDHOBUGA3XFZA4W5DYJIJPYBIB2UQDYFH66SIBOJA, seg_pos={0 0}, upload=499442]: Uploading piece... Dec 29 09:09:24.203281193 (569h40m24.074364238s) [noise=true , address=145.79.227.25:4619 , piece_id=B7SGHLQQ5WSJLFCYC46WKZC45QKPAHXHLDESXVE7ZVJSMRRFER4Q, seg_pos={0 0}, upload=499442]: Uploading piece... Dec 29 09:09:24.208163911 (569h40m24.079246956s) [noise=true , address=45.91.116.8:4045 , piece_id=HNHMEAFWQB55EV22YHNNN263U7EYJ26UA6YBD7HHLFJQR27G5COQ, seg_pos={0 0}, upload=499442]: Uploading piece... Dec 29 09:09:24.210378307 (569h40m24.081461352s) [noise=true , address=77.90.31.11:4555 , piece_id=6LDYNQTC2JOATTJMDYG4XCU3DHU5FRKF5BMHL3SEQ5XFJ6TFNP3A, seg_pos={0 0}, upload=499442]: Uploading piece... Dec 29 09:09:24.213579382 (569h40m24.084662417s) [noise=true , address=77.90.31.19:4841 , piece_id=GACMF3J5TOBKH7DBFJUBLAOVM6NPANPHNSKCGVXYZF6NTX6XWSVA, seg_pos={0 0}, upload=499442]: Uploading piece... Dec 29 09:09:24.215500822 (569h40m24.086583857s) [noise=true , address=77.90.31.10:4519 , piece_id=BO24PMA5FSRNVM7OLQHYGL2TLUJ4PYGOVXWYNOJPW42S7G5HZE3Q, seg_pos={0 0}, upload=499442]: Uploading piece... Dec 29 09:09:24.215973431 (569h40m24.087056476s) [noise=true , address=145.79.227.31:4798 , piece_id=VNADLC736FD5JO4N52QWNGCBHVSDXES2AI3WKPSB4AIQRQRWRL4A, seg_pos={0 0}, upload=499442]: Uploading piece... Dec 29 09:09:24.222012611 (569h40m24.093095657s) [noise=true , address=145.79.227.7:4294 , piece_id=YTPJSCTZLZB2BLC6M5MURG6HAJDQABWFACMQBIRGNE2EBG3T74PQ, seg_pos={0 0}, upload=499442]: Uploading piece... Dec 29 09:09:24.223461410 (569h40m24.094544455s) [noise=true , address=45.91.116.9:4261 , piece_id=BHI5O6QWPVXMIJZMCCV7VEKBVUFXN3E7FONHSDWCSALVRDX2NZ3Q, seg_pos={0 0}, upload=499442]: Uploading piece... Dec 29 09:09:25.905869514 (569h40m25.776952549s) [upload=499442]: Next returned nil segment Dec 29 09:09:25.905882179 (569h40m25.776965224s) [upload=499442]: Waiting for error group managing segments... Dec 29 09:09:26.188279546 (569h40m26.059362581s) [noise=true , address=145.79.227.16:4912 , piece_id=LAW3OSFHRDZ345SKYCFNDJO5MDG4GLAUJ243C54DIOZ34GK543BA, seg_pos={0 0}, upload=499442]: Done uploading piece. err: <nil> Dec 29 09:09:26.195272040 (569h40m26.066355085s) [noise=true , address=145.79.227.13:4804 , piece_id=AAEETYPTHUGKBUKRAKZLTJ6TVDUG2FSGFNIAFOKUKDQKNZWZU4PQ, seg_pos={0 0}, upload=499442]: Done uploading piece. err: <nil> Dec 29 09:09:26.207426815 (569h40m26.07850985s) [noise=true , address=145.79.227.46:4603 , piece_id=4RZPDUGR5CI7IYOIWINGQINDWBEPWQLSWY2T7ZDZY262DV7C5UNQ, seg_pos={0 0}, upload=499442]: Done uploading piece. err: <nil> Dec 29 09:09:26.212434781 (569h40m26.083517816s) [noise=true , address=145.79.227.7:4294 , piece_id=YTPJSCTZLZB2BLC6M5MURG6HAJDQABWFACMQBIRGNE2EBG3T74PQ, seg_pos={0 0}, upload=499442]: Done uploading piece. err: <nil> Dec 29 09:09:26.213308765 (569h40m26.08439181s) [noise=true , address=145.79.227.45:4229 , piece_id=US6NQVN4YBOTBS7GG57BRI7FZ77OHU6WYFMA5BTQKWYHSQFXQNZA, seg_pos={0 0}, upload=499442]: Done uploading piece. err: <nil> Dec 29 09:09:26.215644423 (569h40m26.086727458s) [noise=true , address=145.79.227.31:4798 , piece_id=VNADLC736FD5JO4N52QWNGCBHVSDXES2AI3WKPSB4AIQRQRWRL4A, seg_pos={0 0}, upload=499442]: Done uploading piece. err: <nil> Dec 29 09:09:26.218522473 (569h40m26.089605518s) [noise=true , address=145.79.227.25:4619 , piece_id=B7SGHLQQ5WSJLFCYC46WKZC45QKPAHXHLDESXVE7ZVJSMRRFER4Q, seg_pos={0 0}, upload=499442]: Done uploading piece. err: <nil> Dec 29 09:09:26.380017695 (569h40m26.25110073s) [noise=true , address=77.90.31.19:4841 , piece_id=GACMF3J5TOBKH7DBFJUBLAOVM6NPANPHNSKCGVXYZF6NTX6XWSVA, seg_pos={0 0}, upload=499442]: Done uploading piece. err: <nil> Dec 29 09:09:26.399490054 (569h40m26.270573089s) [noise=true , address=77.90.31.10:4519 , piece_id=BO24PMA5FSRNVM7OLQHYGL2TLUJ4PYGOVXWYNOJPW42S7G5HZE3Q, seg_pos={0 0}, upload=499442]: Done uploading piece. err: <nil> Dec 29 09:09:26.462302348 (569h40m26.333385383s) [noise=true , address=45.91.116.6:4975 , piece_id=QS2DMK6RLXOQGERYYSZGKK6KQLSFUVAQHP7HII6YC7KROYJOBQ5Q, seg_pos={0 0}, upload=499442]: Done uploading piece. err: <nil> Dec 29 09:09:26.497646719 (569h40m26.368729754s) [noise=true , address=77.90.31.11:4555 , piece_id=6LDYNQTC2JOATTJMDYG4XCU3DHU5FRKF5BMHL3SEQ5XFJ6TFNP3A, seg_pos={0 0}, upload=499442]: Done uploading piece. err: <nil> Dec 29 09:09:26.505306996 (569h40m26.376390041s) [noise=true , address=45.91.116.10:4081 , piece_id=PRIHC33CW2ZMDHOBUGA3XFZA4W5DYJIJPYBIB2UQDYFH66SIBOJA, seg_pos={0 0}, upload=499442]: Done uploading piece. err: <nil> Dec 29 09:09:26.538530998 (569h40m26.409614033s) [noise=true , address=45.91.116.12:4226 , piece_id=SZTDTDIAAVRUWK3E27PVYZGYCSC4R3POM2ODTVWWKNIB3T2W7QRQ, seg_pos={0 0}, upload=499442]: Done uploading piece. err: <nil> Dec 29 09:09:26.538554462 (569h40m26.409637497s) [seg_pos={0 0}, upload=499442]: Segment reached optimal threshold of 13 pieces. Dec 29 09:09:26.538758551 (569h40m26.409841596s) [noise=true , address=45.91.116.8:4045 , piece_id=HNHMEAFWQB55EV22YHNNN263U7EYJ26UA6YBD7HHLFJQR27G5COQ, seg_pos={0 0}, upload=499442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 29 09:09:26.538766166 (569h40m26.409849211s) [noise=true , address=45.91.116.8:4045 , piece_id=HNHMEAFWQB55EV22YHNNN263U7EYJ26UA6YBD7HHLFJQR27G5COQ, seg_pos={0 0}, upload=499442]: Long tail cancellation due to optimal threshold Dec 29 09:09:26.539161809 (569h40m26.410244854s) [noise=true , address=45.91.116.9:4261 , piece_id=BHI5O6QWPVXMIJZMCCV7VEKBVUFXN3E7FONHSDWCSALVRDX2NZ3Q, seg_pos={0 0}, upload=499442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:38938->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:38938->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38938->45.91.116.9:4261: use of closed network connection Dec 29 09:09:26.539170345 (569h40m26.410253391s) [noise=true , address=45.91.116.9:4261 , piece_id=BHI5O6QWPVXMIJZMCCV7VEKBVUFXN3E7FONHSDWCSALVRDX2NZ3Q, seg_pos={0 0}, upload=499442]: Long tail cancellation due to optimal threshold Dec 29 09:09:26.539283100 (569h40m26.410366145s) [noise=true , address=77.90.31.14:4662 , piece_id=TTY7Z2MLHY5HWWU6OAQZC7PGJOJCQL2VNKDZZYDEH32D7V2RLWTQ, seg_pos={0 0}, upload=499442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:53050->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:53050->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53050->77.90.31.14:4662: use of closed network connection Dec 29 09:09:26.539290154 (569h40m26.410373199s) [noise=true , address=77.90.31.14:4662 , piece_id=TTY7Z2MLHY5HWWU6OAQZC7PGJOJCQL2VNKDZZYDEH32D7V2RLWTQ, seg_pos={0 0}, upload=499442]: Long tail cancellation due to optimal threshold Dec 29 09:09:26.539308909 (569h40m26.410391954s) [seg_pos={0 0}, upload=499442]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:09:26.539387729 (569h40m26.410470774s) [upload=499442]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:09:26.539393200 (569h40m26.410476245s) [upload=499442]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:09:26.836548429 (569h40m26.707631464s) [upload=499442]: Done uploading segments... Dec 29 09:09:26.836568346 (569h40m26.707651381s) [upload=499442]: Upload finished. err: <nil> Duration: 30.002548905s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [......X............X.......................XX....X..........] duration: 3.37617748s trace: https://jaeger.dev.storj.io/trace/2818f7531409d35c FAIL (2025-12-29 09:09:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.376204261s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 40.543900781s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 09:09:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 40.543927392s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 15.286886202s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 09:09:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 15.286920066s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.......X............X.......................XX....X.........] duration: 3.431930914s trace: https://jaeger.dev.storj.io/trace/2818f7531409d35c FAIL (2025-12-29 09:08:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.431955341s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...X.....XXX......X....X..XXX..X.....X..XX.X...X...XX.X.XX..] duration: 5.990254624s trace: https://jaeger.dev.storj.io/trace/46dfc98662881bd8 FAIL (2025-12-29 09:08:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: KLXG73RMVVBLXFA7OYIMKHCLEAZFOJ7A4HOLDH7JQ47226KQ5EDA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 7H3C3YXRYY67THRZ4V675YMAKYY3ZFKGOTUW25SKM756RDVOPO5Q) context deadline exceeded ======LOGS====== 29 09:07:11.107746943 (569h38m10.978829988s) [upload=499412]: Starting upload Dec 29 09:07:11.107841332 (569h38m10.978924367s) [upload=499412]: Done starting upload Dec 29 09:07:11.107914832 (569h38m10.978997877s) [upload=499412]: Uploading segments... Dec 29 09:07:11.108063205 (569h38m10.97914624s) [upload=499412]: Got next segment. Inline: false Dec 29 09:07:11.108123590 (569h38m10.979206635s) [upload=499412]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:07:11.108132387 (569h38m10.979215422s) [upload=499412]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:07:11.356121836 (569h38m11.227204871s) [seg_pos={0 0}, upload=499412]: Begin upload segment... Dec 29 09:07:11.356184034 (569h38m11.22726708s) [seg_pos={0 0}, upload=499412]: Done begin upload segment. Dec 29 09:07:11.356216857 (569h38m11.227299892s) [noise=true , address=77.90.31.13:4629 , piece_id=7PIETJD5KVPCR3FKXQ4XNQQDMC56BO7HZTZWCP2P6Z2BBUSCN7KA, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.356269867 (569h38m11.227352913s) [noise=true , address=145.79.227.48:4302 , piece_id=BBBCEVIQRAHHO65R2CMQYHTB6JLNKQMLGTNTYKSZGZSFQIK6UTNA, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.356931467 (569h38m11.228014512s) [noise=true , address=45.91.116.13:4155 , piece_id=JL6XYLGPGMMJUGBU7BWJYTKTAWJOSNSBKQBIR5AQWP24NYZDA6AQ, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.357378669 (569h38m11.228461714s) [noise=true , address=77.90.31.18:4805 , piece_id=KLXG73RMVVBLXFA7OYIMKHCLEAZFOJ7A4HOLDH7JQ47226KQ5EDA, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.358019328 (569h38m11.229102374s) [noise=true , address=77.90.31.10:4517 , piece_id=5HC3KW5T52FVSCQXE322UQWEQHHUCZTAYJTMCJMWMXROHH2BLN2A, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.358631505 (569h38m11.22971454s) [noise=true , address=145.79.227.6:4258 , piece_id=VUCY2SYVGC2XRSMGQG7DH5JLOLPGZKVGX6DCFN3KMABNYM6FQITA, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.359395580 (569h38m11.230478625s) [noise=true , address=45.91.116.11:4117 , piece_id=QLZZJDP5ATWV6PIAVLAALDUWGBKQ37AH6ZVIXPPNZXHU3J6VB7WA, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.359669561 (569h38m11.230752606s) [noise=true , address=145.79.227.5:4221 , piece_id=5WNO3BOTFARORU4FXK5EF5DTC27DR4G65QPIEPEMNYEG6NN56P2Q, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.360477079 (569h38m11.231560124s) [noise=true , address=45.91.116.10:4084 , piece_id=7H3C3YXRYY67THRZ4V675YMAKYY3ZFKGOTUW25SKM756RDVOPO5Q, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.360870297 (569h38m11.231953342s) [noise=true , address=145.79.227.19:4077 , piece_id=QQSHXZK4JKFSAF4X5CLZX4EI6P37AJ4XMT2OWAPTDDUJXJQVBLGQ, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.361671493 (569h38m11.232754538s) [noise=true , address=145.79.227.11:4729 , piece_id=I4PSZASP4AEONVTUZPGAHP7WFIH7VD55TZK7DNDNKEAICAX4BPOA, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.361940205 (569h38m11.23302325s) [noise=true , address=145.79.227.14:4837 , piece_id=23C5K344ALE44YJEUORHJVL4MNPGLH54VCFSNYGFZKBBZYLKSHLA, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.362539927 (569h38m11.233622972s) [noise=true , address=145.79.227.39:4014 , piece_id=24JGJLE476JUKIFOZM4NE4NCGOHIRNH4Y3YLUW64PI5OKSFUZMZQ, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.362752802 (569h38m11.233835837s) [noise=true , address=145.79.227.47:4267 , piece_id=FYY246DM6IBPPL3XY277IQXNKS375XGSP6AUQOKINB3RR6ZTIDHA, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.363126644 (569h38m11.234209679s) [noise=true , address=145.79.227.3:4151 , piece_id=YSLVFEEBADOHYDADWZKWJFZV5GNPVBILORQDQSRGZMMLMNZ2ROHQ, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.364786224 (569h38m11.235869269s) [noise=true , address=145.79.227.26:4655 , piece_id=ZBIQRGOZ5SBAGJXQJXLHI2HB7D7YELCXYYDBSHJZLV5IQJY2JQ7A, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:13.064128894 (569h38m12.935211929s) [upload=499412]: Next returned nil segment Dec 29 09:07:13.064188207 (569h38m12.935271252s) [upload=499412]: Waiting for error group managing segments... Dec 29 09:07:13.347381629 (569h38m13.218464664s) [noise=true , address=145.79.227.14:4837 , piece_id=23C5K344ALE44YJEUORHJVL4MNPGLH54VCFSNYGFZKBBZYLKSHLA, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.347783874 (569h38m13.218866919s) [noise=true , address=145.79.227.11:4729 , piece_id=I4PSZASP4AEONVTUZPGAHP7WFIH7VD55TZK7DNDNKEAICAX4BPOA, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.352119751 (569h38m13.223202796s) [noise=true , address=145.79.227.48:4302 , piece_id=BBBCEVIQRAHHO65R2CMQYHTB6JLNKQMLGTNTYKSZGZSFQIK6UTNA, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.355979982 (569h38m13.227063017s) [noise=true , address=145.79.227.6:4258 , piece_id=VUCY2SYVGC2XRSMGQG7DH5JLOLPGZKVGX6DCFN3KMABNYM6FQITA, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.359169234 (569h38m13.230252279s) [noise=true , address=145.79.227.19:4077 , piece_id=QQSHXZK4JKFSAF4X5CLZX4EI6P37AJ4XMT2OWAPTDDUJXJQVBLGQ, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.364526645 (569h38m13.23560968s) [noise=true , address=145.79.227.47:4267 , piece_id=FYY246DM6IBPPL3XY277IQXNKS375XGSP6AUQOKINB3RR6ZTIDHA, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.366246951 (569h38m13.237329996s) [noise=true , address=145.79.227.26:4655 , piece_id=ZBIQRGOZ5SBAGJXQJXLHI2HB7D7YELCXYYDBSHJZLV5IQJY2JQ7A, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.373256127 (569h38m13.244339162s) [noise=true , address=145.79.227.3:4151 , piece_id=YSLVFEEBADOHYDADWZKWJFZV5GNPVBILORQDQSRGZMMLMNZ2ROHQ, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.380394039 (569h38m13.251477084s) [noise=true , address=145.79.227.5:4221 , piece_id=5WNO3BOTFARORU4FXK5EF5DTC27DR4G65QPIEPEMNYEG6NN56P2Q, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.381647385 (569h38m13.25273043s) [noise=true , address=145.79.227.39:4014 , piece_id=24JGJLE476JUKIFOZM4NE4NCGOHIRNH4Y3YLUW64PI5OKSFUZMZQ, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.535547776 (569h38m13.406630811s) [noise=true , address=77.90.31.18:4805 , piece_id=KLXG73RMVVBLXFA7OYIMKHCLEAZFOJ7A4HOLDH7JQ47226KQ5EDA, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.614690084 (569h38m13.485773119s) [noise=true , address=45.91.116.11:4117 , piece_id=QLZZJDP5ATWV6PIAVLAALDUWGBKQ37AH6ZVIXPPNZXHU3J6VB7WA, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.615360472 (569h38m13.486443507s) [noise=true , address=45.91.116.10:4084 , piece_id=7H3C3YXRYY67THRZ4V675YMAKYY3ZFKGOTUW25SKM756RDVOPO5Q, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.615371072 (569h38m13.486454107s) [seg_pos={0 0}, upload=499412]: Segment reached optimal threshold of 13 pieces. Dec 29 09:07:13.615583546 (569h38m13.486666581s) [noise=true , address=45.91.116.13:4155 , piece_id=JL6XYLGPGMMJUGBU7BWJYTKTAWJOSNSBKQBIR5AQWP24NYZDA6AQ, seg_pos={0 0}, upload=499412]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 09:07:13.615591481 (569h38m13.486674526s) [noise=true , address=45.91.116.13:4155 , piece_id=JL6XYLGPGMMJUGBU7BWJYTKTAWJOSNSBKQBIR5AQWP24NYZDA6AQ, seg_pos={0 0}, upload=499412]: Long tail cancellation due to optimal threshold Dec 29 09:07:13.615852599 (569h38m13.486935634s) [noise=true , address=77.90.31.13:4629 , piece_id=7PIETJD5KVPCR3FKXQ4XNQQDMC56BO7HZTZWCP2P6Z2BBUSCN7KA, seg_pos={0 0}, upload=499412]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:44936->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:44936->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44936->77.90.31.13:4629: use of closed network connection Dec 29 09:07:13.615859752 (569h38m13.486942797s) [noise=true , address=77.90.31.13:4629 , piece_id=7PIETJD5KVPCR3FKXQ4XNQQDMC56BO7HZTZWCP2P6Z2BBUSCN7KA, seg_pos={0 0}, upload=499412]: Long tail cancellation due to optimal threshold Dec 29 09:07:13.616237011 (569h38m13.487320056s) [noise=true , address=77.90.31.10:4517 , piece_id=5HC3KW5T52FVSCQXE322UQWEQHHUCZTAYJTMCJMWMXROHH2BLN2A, seg_pos={0 0}, upload=499412]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 29 09:07:13.616257469 (569h38m13.487340514s) [noise=true , address=77.90.31.10:4517 , piece_id=5HC3KW5T52FVSCQXE322UQWEQHHUCZTAYJTMCJMWMXROHH2BLN2A, seg_pos={0 0}, upload=499412]: Long tail cancellation due to optimal threshold Dec 29 09:07:13.616309038 (569h38m13.487392083s) [seg_pos={0 0}, upload=499412]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:07:13.616464283 (569h38m13.487547328s) [upload=499412]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:07:13.616472618 (569h38m13.487555653s) [upload=499412]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:07:13.904989319 (569h38m13.776072374s) [upload=499412]: Done uploading segments... Dec 29 09:07:13.905014768 (569h38m13.776097823s) [upload=499412]: Upload finished. err: <nil> Duration: 5.990280163s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 46.937258571s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 09:08:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 46.937288629s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X...X.....XXX......X....X..XXX..X.....X..XX.X...X...XX.X.XX.] duration: 7.65781238s trace: https://jaeger.dev.storj.io/trace/46dfc98662881bd8 FAIL (2025-12-29 09:07:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: KLXG73RMVVBLXFA7OYIMKHCLEAZFOJ7A4HOLDH7JQ47226KQ5EDA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 7H3C3YXRYY67THRZ4V675YMAKYY3ZFKGOTUW25SKM756RDVOPO5Q) context deadline exceeded ======LOGS====== 29 09:07:11.107746943 (569h38m10.978829988s) [upload=499412]: Starting upload Dec 29 09:07:11.107841332 (569h38m10.978924367s) [upload=499412]: Done starting upload Dec 29 09:07:11.107914832 (569h38m10.978997877s) [upload=499412]: Uploading segments... Dec 29 09:07:11.108063205 (569h38m10.97914624s) [upload=499412]: Got next segment. Inline: false Dec 29 09:07:11.108123590 (569h38m10.979206635s) [upload=499412]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:07:11.108132387 (569h38m10.979215422s) [upload=499412]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:07:11.356121836 (569h38m11.227204871s) [seg_pos={0 0}, upload=499412]: Begin upload segment... Dec 29 09:07:11.356184034 (569h38m11.22726708s) [seg_pos={0 0}, upload=499412]: Done begin upload segment. Dec 29 09:07:11.356216857 (569h38m11.227299892s) [noise=true , address=77.90.31.13:4629 , piece_id=7PIETJD5KVPCR3FKXQ4XNQQDMC56BO7HZTZWCP2P6Z2BBUSCN7KA, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.356269867 (569h38m11.227352913s) [noise=true , address=145.79.227.48:4302 , piece_id=BBBCEVIQRAHHO65R2CMQYHTB6JLNKQMLGTNTYKSZGZSFQIK6UTNA, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.356931467 (569h38m11.228014512s) [noise=true , address=45.91.116.13:4155 , piece_id=JL6XYLGPGMMJUGBU7BWJYTKTAWJOSNSBKQBIR5AQWP24NYZDA6AQ, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.357378669 (569h38m11.228461714s) [noise=true , address=77.90.31.18:4805 , piece_id=KLXG73RMVVBLXFA7OYIMKHCLEAZFOJ7A4HOLDH7JQ47226KQ5EDA, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.358019328 (569h38m11.229102374s) [noise=true , address=77.90.31.10:4517 , piece_id=5HC3KW5T52FVSCQXE322UQWEQHHUCZTAYJTMCJMWMXROHH2BLN2A, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.358631505 (569h38m11.22971454s) [noise=true , address=145.79.227.6:4258 , piece_id=VUCY2SYVGC2XRSMGQG7DH5JLOLPGZKVGX6DCFN3KMABNYM6FQITA, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.359395580 (569h38m11.230478625s) [noise=true , address=45.91.116.11:4117 , piece_id=QLZZJDP5ATWV6PIAVLAALDUWGBKQ37AH6ZVIXPPNZXHU3J6VB7WA, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.359669561 (569h38m11.230752606s) [noise=true , address=145.79.227.5:4221 , piece_id=5WNO3BOTFARORU4FXK5EF5DTC27DR4G65QPIEPEMNYEG6NN56P2Q, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.360477079 (569h38m11.231560124s) [noise=true , address=45.91.116.10:4084 , piece_id=7H3C3YXRYY67THRZ4V675YMAKYY3ZFKGOTUW25SKM756RDVOPO5Q, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.360870297 (569h38m11.231953342s) [noise=true , address=145.79.227.19:4077 , piece_id=QQSHXZK4JKFSAF4X5CLZX4EI6P37AJ4XMT2OWAPTDDUJXJQVBLGQ, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.361671493 (569h38m11.232754538s) [noise=true , address=145.79.227.11:4729 , piece_id=I4PSZASP4AEONVTUZPGAHP7WFIH7VD55TZK7DNDNKEAICAX4BPOA, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.361940205 (569h38m11.23302325s) [noise=true , address=145.79.227.14:4837 , piece_id=23C5K344ALE44YJEUORHJVL4MNPGLH54VCFSNYGFZKBBZYLKSHLA, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.362539927 (569h38m11.233622972s) [noise=true , address=145.79.227.39:4014 , piece_id=24JGJLE476JUKIFOZM4NE4NCGOHIRNH4Y3YLUW64PI5OKSFUZMZQ, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.362752802 (569h38m11.233835837s) [noise=true , address=145.79.227.47:4267 , piece_id=FYY246DM6IBPPL3XY277IQXNKS375XGSP6AUQOKINB3RR6ZTIDHA, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.363126644 (569h38m11.234209679s) [noise=true , address=145.79.227.3:4151 , piece_id=YSLVFEEBADOHYDADWZKWJFZV5GNPVBILORQDQSRGZMMLMNZ2ROHQ, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.364786224 (569h38m11.235869269s) [noise=true , address=145.79.227.26:4655 , piece_id=ZBIQRGOZ5SBAGJXQJXLHI2HB7D7YELCXYYDBSHJZLV5IQJY2JQ7A, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:13.064128894 (569h38m12.935211929s) [upload=499412]: Next returned nil segment Dec 29 09:07:13.064188207 (569h38m12.935271252s) [upload=499412]: Waiting for error group managing segments... Dec 29 09:07:13.347381629 (569h38m13.218464664s) [noise=true , address=145.79.227.14:4837 , piece_id=23C5K344ALE44YJEUORHJVL4MNPGLH54VCFSNYGFZKBBZYLKSHLA, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.347783874 (569h38m13.218866919s) [noise=true , address=145.79.227.11:4729 , piece_id=I4PSZASP4AEONVTUZPGAHP7WFIH7VD55TZK7DNDNKEAICAX4BPOA, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.352119751 (569h38m13.223202796s) [noise=true , address=145.79.227.48:4302 , piece_id=BBBCEVIQRAHHO65R2CMQYHTB6JLNKQMLGTNTYKSZGZSFQIK6UTNA, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.355979982 (569h38m13.227063017s) [noise=true , address=145.79.227.6:4258 , piece_id=VUCY2SYVGC2XRSMGQG7DH5JLOLPGZKVGX6DCFN3KMABNYM6FQITA, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.359169234 (569h38m13.230252279s) [noise=true , address=145.79.227.19:4077 , piece_id=QQSHXZK4JKFSAF4X5CLZX4EI6P37AJ4XMT2OWAPTDDUJXJQVBLGQ, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.364526645 (569h38m13.23560968s) [noise=true , address=145.79.227.47:4267 , piece_id=FYY246DM6IBPPL3XY277IQXNKS375XGSP6AUQOKINB3RR6ZTIDHA, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.366246951 (569h38m13.237329996s) [noise=true , address=145.79.227.26:4655 , piece_id=ZBIQRGOZ5SBAGJXQJXLHI2HB7D7YELCXYYDBSHJZLV5IQJY2JQ7A, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.373256127 (569h38m13.244339162s) [noise=true , address=145.79.227.3:4151 , piece_id=YSLVFEEBADOHYDADWZKWJFZV5GNPVBILORQDQSRGZMMLMNZ2ROHQ, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.380394039 (569h38m13.251477084s) [noise=true , address=145.79.227.5:4221 , piece_id=5WNO3BOTFARORU4FXK5EF5DTC27DR4G65QPIEPEMNYEG6NN56P2Q, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.381647385 (569h38m13.25273043s) [noise=true , address=145.79.227.39:4014 , piece_id=24JGJLE476JUKIFOZM4NE4NCGOHIRNH4Y3YLUW64PI5OKSFUZMZQ, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.535547776 (569h38m13.406630811s) [noise=true , address=77.90.31.18:4805 , piece_id=KLXG73RMVVBLXFA7OYIMKHCLEAZFOJ7A4HOLDH7JQ47226KQ5EDA, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.614690084 (569h38m13.485773119s) [noise=true , address=45.91.116.11:4117 , piece_id=QLZZJDP5ATWV6PIAVLAALDUWGBKQ37AH6ZVIXPPNZXHU3J6VB7WA, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.615360472 (569h38m13.486443507s) [noise=true , address=45.91.116.10:4084 , piece_id=7H3C3YXRYY67THRZ4V675YMAKYY3ZFKGOTUW25SKM756RDVOPO5Q, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.615371072 (569h38m13.486454107s) [seg_pos={0 0}, upload=499412]: Segment reached optimal threshold of 13 pieces. Dec 29 09:07:13.615583546 (569h38m13.486666581s) [noise=true , address=45.91.116.13:4155 , piece_id=JL6XYLGPGMMJUGBU7BWJYTKTAWJOSNSBKQBIR5AQWP24NYZDA6AQ, seg_pos={0 0}, upload=499412]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 09:07:13.615591481 (569h38m13.486674526s) [noise=true , address=45.91.116.13:4155 , piece_id=JL6XYLGPGMMJUGBU7BWJYTKTAWJOSNSBKQBIR5AQWP24NYZDA6AQ, seg_pos={0 0}, upload=499412]: Long tail cancellation due to optimal threshold Dec 29 09:07:13.615852599 (569h38m13.486935634s) [noise=true , address=77.90.31.13:4629 , piece_id=7PIETJD5KVPCR3FKXQ4XNQQDMC56BO7HZTZWCP2P6Z2BBUSCN7KA, seg_pos={0 0}, upload=499412]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:44936->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:44936->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44936->77.90.31.13:4629: use of closed network connection Dec 29 09:07:13.615859752 (569h38m13.486942797s) [noise=true , address=77.90.31.13:4629 , piece_id=7PIETJD5KVPCR3FKXQ4XNQQDMC56BO7HZTZWCP2P6Z2BBUSCN7KA, seg_pos={0 0}, upload=499412]: Long tail cancellation due to optimal threshold Dec 29 09:07:13.616237011 (569h38m13.487320056s) [noise=true , address=77.90.31.10:4517 , piece_id=5HC3KW5T52FVSCQXE322UQWEQHHUCZTAYJTMCJMWMXROHH2BLN2A, seg_pos={0 0}, upload=499412]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 29 09:07:13.616257469 (569h38m13.487340514s) [noise=true , address=77.90.31.10:4517 , piece_id=5HC3KW5T52FVSCQXE322UQWEQHHUCZTAYJTMCJMWMXROHH2BLN2A, seg_pos={0 0}, upload=499412]: Long tail cancellation due to optimal threshold Dec 29 09:07:13.616309038 (569h38m13.487392083s) [seg_pos={0 0}, upload=499412]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:07:13.616464283 (569h38m13.487547328s) [upload=499412]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:07:13.616472618 (569h38m13.487555653s) [upload=499412]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:07:13.904989319 (569h38m13.776072374s) [upload=499412]: Done uploading segments... Dec 29 09:07:13.905014768 (569h38m13.776097823s) [upload=499412]: Upload finished. err: <nil> Duration: 7.657842337s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [X.......X............X.......................XX....X........] duration: 3.481590873s trace: https://jaeger.dev.storj.io/trace/2818f7531409d35c FAIL (2025-12-29 09:07:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.481619078s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X...X.....XXX......X....X..XXX..X.....X..XX.X...X...XX.X.XX] duration: 30.0031193s trace: https://jaeger.dev.storj.io/trace/46dfc98662881bd8 FAIL (2025-12-29 09:07:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: KLXG73RMVVBLXFA7OYIMKHCLEAZFOJ7A4HOLDH7JQ47226KQ5EDA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 7H3C3YXRYY67THRZ4V675YMAKYY3ZFKGOTUW25SKM756RDVOPO5Q) context deadline exceeded ======LOGS====== 29 09:07:11.107746943 (569h38m10.978829988s) [upload=499412]: Starting upload Dec 29 09:07:11.107841332 (569h38m10.978924367s) [upload=499412]: Done starting upload Dec 29 09:07:11.107914832 (569h38m10.978997877s) [upload=499412]: Uploading segments... Dec 29 09:07:11.108063205 (569h38m10.97914624s) [upload=499412]: Got next segment. Inline: false Dec 29 09:07:11.108123590 (569h38m10.979206635s) [upload=499412]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:07:11.108132387 (569h38m10.979215422s) [upload=499412]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:07:11.356121836 (569h38m11.227204871s) [seg_pos={0 0}, upload=499412]: Begin upload segment... Dec 29 09:07:11.356184034 (569h38m11.22726708s) [seg_pos={0 0}, upload=499412]: Done begin upload segment. Dec 29 09:07:11.356216857 (569h38m11.227299892s) [noise=true , address=77.90.31.13:4629 , piece_id=7PIETJD5KVPCR3FKXQ4XNQQDMC56BO7HZTZWCP2P6Z2BBUSCN7KA, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.356269867 (569h38m11.227352913s) [noise=true , address=145.79.227.48:4302 , piece_id=BBBCEVIQRAHHO65R2CMQYHTB6JLNKQMLGTNTYKSZGZSFQIK6UTNA, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.356931467 (569h38m11.228014512s) [noise=true , address=45.91.116.13:4155 , piece_id=JL6XYLGPGMMJUGBU7BWJYTKTAWJOSNSBKQBIR5AQWP24NYZDA6AQ, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.357378669 (569h38m11.228461714s) [noise=true , address=77.90.31.18:4805 , piece_id=KLXG73RMVVBLXFA7OYIMKHCLEAZFOJ7A4HOLDH7JQ47226KQ5EDA, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.358019328 (569h38m11.229102374s) [noise=true , address=77.90.31.10:4517 , piece_id=5HC3KW5T52FVSCQXE322UQWEQHHUCZTAYJTMCJMWMXROHH2BLN2A, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.358631505 (569h38m11.22971454s) [noise=true , address=145.79.227.6:4258 , piece_id=VUCY2SYVGC2XRSMGQG7DH5JLOLPGZKVGX6DCFN3KMABNYM6FQITA, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.359395580 (569h38m11.230478625s) [noise=true , address=45.91.116.11:4117 , piece_id=QLZZJDP5ATWV6PIAVLAALDUWGBKQ37AH6ZVIXPPNZXHU3J6VB7WA, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.359669561 (569h38m11.230752606s) [noise=true , address=145.79.227.5:4221 , piece_id=5WNO3BOTFARORU4FXK5EF5DTC27DR4G65QPIEPEMNYEG6NN56P2Q, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.360477079 (569h38m11.231560124s) [noise=true , address=45.91.116.10:4084 , piece_id=7H3C3YXRYY67THRZ4V675YMAKYY3ZFKGOTUW25SKM756RDVOPO5Q, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.360870297 (569h38m11.231953342s) [noise=true , address=145.79.227.19:4077 , piece_id=QQSHXZK4JKFSAF4X5CLZX4EI6P37AJ4XMT2OWAPTDDUJXJQVBLGQ, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.361671493 (569h38m11.232754538s) [noise=true , address=145.79.227.11:4729 , piece_id=I4PSZASP4AEONVTUZPGAHP7WFIH7VD55TZK7DNDNKEAICAX4BPOA, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.361940205 (569h38m11.23302325s) [noise=true , address=145.79.227.14:4837 , piece_id=23C5K344ALE44YJEUORHJVL4MNPGLH54VCFSNYGFZKBBZYLKSHLA, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.362539927 (569h38m11.233622972s) [noise=true , address=145.79.227.39:4014 , piece_id=24JGJLE476JUKIFOZM4NE4NCGOHIRNH4Y3YLUW64PI5OKSFUZMZQ, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.362752802 (569h38m11.233835837s) [noise=true , address=145.79.227.47:4267 , piece_id=FYY246DM6IBPPL3XY277IQXNKS375XGSP6AUQOKINB3RR6ZTIDHA, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.363126644 (569h38m11.234209679s) [noise=true , address=145.79.227.3:4151 , piece_id=YSLVFEEBADOHYDADWZKWJFZV5GNPVBILORQDQSRGZMMLMNZ2ROHQ, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:11.364786224 (569h38m11.235869269s) [noise=true , address=145.79.227.26:4655 , piece_id=ZBIQRGOZ5SBAGJXQJXLHI2HB7D7YELCXYYDBSHJZLV5IQJY2JQ7A, seg_pos={0 0}, upload=499412]: Uploading piece... Dec 29 09:07:13.064128894 (569h38m12.935211929s) [upload=499412]: Next returned nil segment Dec 29 09:07:13.064188207 (569h38m12.935271252s) [upload=499412]: Waiting for error group managing segments... Dec 29 09:07:13.347381629 (569h38m13.218464664s) [noise=true , address=145.79.227.14:4837 , piece_id=23C5K344ALE44YJEUORHJVL4MNPGLH54VCFSNYGFZKBBZYLKSHLA, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.347783874 (569h38m13.218866919s) [noise=true , address=145.79.227.11:4729 , piece_id=I4PSZASP4AEONVTUZPGAHP7WFIH7VD55TZK7DNDNKEAICAX4BPOA, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.352119751 (569h38m13.223202796s) [noise=true , address=145.79.227.48:4302 , piece_id=BBBCEVIQRAHHO65R2CMQYHTB6JLNKQMLGTNTYKSZGZSFQIK6UTNA, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.355979982 (569h38m13.227063017s) [noise=true , address=145.79.227.6:4258 , piece_id=VUCY2SYVGC2XRSMGQG7DH5JLOLPGZKVGX6DCFN3KMABNYM6FQITA, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.359169234 (569h38m13.230252279s) [noise=true , address=145.79.227.19:4077 , piece_id=QQSHXZK4JKFSAF4X5CLZX4EI6P37AJ4XMT2OWAPTDDUJXJQVBLGQ, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.364526645 (569h38m13.23560968s) [noise=true , address=145.79.227.47:4267 , piece_id=FYY246DM6IBPPL3XY277IQXNKS375XGSP6AUQOKINB3RR6ZTIDHA, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.366246951 (569h38m13.237329996s) [noise=true , address=145.79.227.26:4655 , piece_id=ZBIQRGOZ5SBAGJXQJXLHI2HB7D7YELCXYYDBSHJZLV5IQJY2JQ7A, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.373256127 (569h38m13.244339162s) [noise=true , address=145.79.227.3:4151 , piece_id=YSLVFEEBADOHYDADWZKWJFZV5GNPVBILORQDQSRGZMMLMNZ2ROHQ, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.380394039 (569h38m13.251477084s) [noise=true , address=145.79.227.5:4221 , piece_id=5WNO3BOTFARORU4FXK5EF5DTC27DR4G65QPIEPEMNYEG6NN56P2Q, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.381647385 (569h38m13.25273043s) [noise=true , address=145.79.227.39:4014 , piece_id=24JGJLE476JUKIFOZM4NE4NCGOHIRNH4Y3YLUW64PI5OKSFUZMZQ, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.535547776 (569h38m13.406630811s) [noise=true , address=77.90.31.18:4805 , piece_id=KLXG73RMVVBLXFA7OYIMKHCLEAZFOJ7A4HOLDH7JQ47226KQ5EDA, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.614690084 (569h38m13.485773119s) [noise=true , address=45.91.116.11:4117 , piece_id=QLZZJDP5ATWV6PIAVLAALDUWGBKQ37AH6ZVIXPPNZXHU3J6VB7WA, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.615360472 (569h38m13.486443507s) [noise=true , address=45.91.116.10:4084 , piece_id=7H3C3YXRYY67THRZ4V675YMAKYY3ZFKGOTUW25SKM756RDVOPO5Q, seg_pos={0 0}, upload=499412]: Done uploading piece. err: <nil> Dec 29 09:07:13.615371072 (569h38m13.486454107s) [seg_pos={0 0}, upload=499412]: Segment reached optimal threshold of 13 pieces. Dec 29 09:07:13.615583546 (569h38m13.486666581s) [noise=true , address=45.91.116.13:4155 , piece_id=JL6XYLGPGMMJUGBU7BWJYTKTAWJOSNSBKQBIR5AQWP24NYZDA6AQ, seg_pos={0 0}, upload=499412]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 09:07:13.615591481 (569h38m13.486674526s) [noise=true , address=45.91.116.13:4155 , piece_id=JL6XYLGPGMMJUGBU7BWJYTKTAWJOSNSBKQBIR5AQWP24NYZDA6AQ, seg_pos={0 0}, upload=499412]: Long tail cancellation due to optimal threshold Dec 29 09:07:13.615852599 (569h38m13.486935634s) [noise=true , address=77.90.31.13:4629 , piece_id=7PIETJD5KVPCR3FKXQ4XNQQDMC56BO7HZTZWCP2P6Z2BBUSCN7KA, seg_pos={0 0}, upload=499412]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:44936->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:44936->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44936->77.90.31.13:4629: use of closed network connection Dec 29 09:07:13.615859752 (569h38m13.486942797s) [noise=true , address=77.90.31.13:4629 , piece_id=7PIETJD5KVPCR3FKXQ4XNQQDMC56BO7HZTZWCP2P6Z2BBUSCN7KA, seg_pos={0 0}, upload=499412]: Long tail cancellation due to optimal threshold Dec 29 09:07:13.616237011 (569h38m13.487320056s) [noise=true , address=77.90.31.10:4517 , piece_id=5HC3KW5T52FVSCQXE322UQWEQHHUCZTAYJTMCJMWMXROHH2BLN2A, seg_pos={0 0}, upload=499412]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 29 09:07:13.616257469 (569h38m13.487340514s) [noise=true , address=77.90.31.10:4517 , piece_id=5HC3KW5T52FVSCQXE322UQWEQHHUCZTAYJTMCJMWMXROHH2BLN2A, seg_pos={0 0}, upload=499412]: Long tail cancellation due to optimal threshold Dec 29 09:07:13.616309038 (569h38m13.487392083s) [seg_pos={0 0}, upload=499412]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:07:13.616464283 (569h38m13.487547328s) [upload=499412]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:07:13.616472618 (569h38m13.487555653s) [upload=499412]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:07:13.904989319 (569h38m13.776072374s) [upload=499412]: Done uploading segments... Dec 29 09:07:13.905014768 (569h38m13.776097823s) [upload=499412]: Upload finished. err: <nil> Duration: 30.003137615s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 31.730747227s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 09:07:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 31.730778256s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.X.......X............X.......................XX....X.......] duration: 3.920677122s trace: https://jaeger.dev.storj.io/trace/2818f7531409d35c FAIL (2025-12-29 09:06:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.920698353s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..X...X.....XXX......X....X..XXX..X.....X..XX.X...X...XX.X.X] duration: 30.003199928s trace: https://jaeger.dev.storj.io/trace/2291def68e6aedd4 FAIL (2025-12-29 09:06:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: IY45BWYJXV5RLWQGKZVMFO3NSVKHKC46ZNXUU4F4GTZE5TNDGLPA) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: POLYHLZNIWON6UYDXQMFGKNUCIYBUMYMZCQUOOQCH45CRAPFCEKQ) context deadline exceeded ======LOGS====== Dec 29 09:06:17.668584110 (569h37m17.539667155s) [upload=499401]: Starting upload Dec 29 09:06:17.668647239 (569h37m17.539730275s) [upload=499401]: Done starting upload Dec 29 09:06:17.668739395 (569h37m17.53982244s) [upload=499401]: Uploading segments... Dec 29 09:06:17.668865656 (569h37m17.539948701s) [upload=499401]: Got next segment. Inline: false Dec 29 09:06:17.668912826 (569h37m17.539995871s) [upload=499401]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:06:17.668919598 (569h37m17.540002643s) [upload=499401]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:06:17.915676632 (569h37m17.786759677s) [seg_pos={0 0}, upload=499401]: Begin upload segment... Dec 29 09:06:17.915723481 (569h37m17.786806516s) [seg_pos={0 0}, upload=499401]: Done begin upload segment. Dec 29 09:06:17.915755221 (569h37m17.786838256s) [noise=true , address=77.90.31.19:4841 , piece_id=POLYHLZNIWON6UYDXQMFGKNUCIYBUMYMZCQUOOQCH45CRAPFCEKQ, seg_pos={0 0}, upload=499401]: Uploading piece... Dec 29 09:06:17.915814064 (569h37m17.786897099s) [noise=true , address=145.79.227.45:4230 , piece_id=NGUPSZ57D7VFGTDGJG7TBPAGZR3RTZ6NRW2KVL2ABSHJFBQPW4RA, seg_pos={0 0}, upload=499401]: Uploading piece... Dec 29 09:06:17.916412543 (569h37m17.787495578s) [noise=true , address=145.79.227.39:4015 , piece_id=U6S7MRFZIMYU23UDY6AACJLGNA4ZFDEYZOMIHDR5TSIM5VQ4X74A, seg_pos={0 0}, upload=499401]: Uploading piece... Dec 29 09:06:17.916557879 (569h37m17.787640915s) [noise=true , address=145.79.227.7:4295 , piece_id=RPACOVOHGZVJ6HTKB6WMQQQSP47U2SYCRU7Y4V6TL7XKKKCQZN7Q, seg_pos={0 0}, upload=499401]: Uploading piece... Dec 29 09:06:17.916778369 (569h37m17.787861404s) [noise=true , address=145.79.227.47:4265 , piece_id=X5CJZ5ANYB3FDQSGUNRD4IDZARZTH5UEWE73OYE7YH5HZ7YOZP7Q, seg_pos={0 0}, upload=499401]: Uploading piece... Dec 29 09:06:17.917010782 (569h37m17.788093817s) [noise=true , address=77.90.31.12:4590 , piece_id=RSTOIMLZRTPMYTM4YJWFTP66EFBL3PAMSDSQ3UTQLCTHFE62MSMA, seg_pos={0 0}, upload=499401]: Uploading piece... Dec 29 09:06:17.917122374 (569h37m17.788205409s) [noise=true , address=145.79.227.34:4906 , piece_id=AAEC7LJK7ZDM3T66EKUUFCV7HLSVCMJXBGM7VC7JBEKZ3GCWXKPA, seg_pos={0 0}, upload=499401]: Uploading piece... Dec 29 09:06:17.917453325 (569h37m17.78853636s) [noise=true , address=45.91.116.8:4046 , piece_id=7ZI3WICEE3CYLYNKCSG2JR6PIXPIXDULUTFKJW2HISSN4KE754XA, seg_pos={0 0}, upload=499401]: Uploading piece... Dec 29 09:06:17.917525051 (569h37m17.788608096s) [noise=true , address=145.79.227.35:4483 , piece_id=SH255JUSIEBQNCYEQWYK27LSNTLWKZXU2SA64ZS4KTVBSNLK7UAQ, seg_pos={0 0}, upload=499401]: Uploading piece... Dec 29 09:06:17.917902871 (569h37m17.788985906s) [noise=true , address=145.79.227.13:4802 , piece_id=26VYYSIDI62FHCDUR4PEUN4FRV7FWBVI3S2JFQIE4AEPC5AWYOHA, seg_pos={0 0}, upload=499401]: Uploading piece... Dec 29 09:06:17.918064869 (569h37m17.789147904s) [noise=true , address=145.79.227.52:4445 , piece_id=N7GOJLI7A4Q5ZS6IWZHZKIL2MO4ZOT5ZYNGAAD2AO53KDIKTHOPA, seg_pos={0 0}, upload=499401]: Uploading piece... Dec 29 09:06:17.918238990 (569h37m17.789322035s) [noise=true , address=77.90.31.21:4877 , piece_id=IY45BWYJXV5RLWQGKZVMFO3NSVKHKC46ZNXUU4F4GTZE5TNDGLPA, seg_pos={0 0}, upload=499401]: Uploading piece... Dec 29 09:06:17.918552547 (569h37m17.789635582s) [noise=true , address=45.91.116.13:4154 , piece_id=SBP7IEJE7QCMN5KPVXVMNS2JF5TM23DSKM7DPQGLSRIEYAGNJRUA, seg_pos={0 0}, upload=499401]: Uploading piece... Dec 29 09:06:17.918660854 (569h37m17.789743889s) [noise=true , address=145.79.227.4:4185 , piece_id=TJZUBWWCSOU7L5C6XWOX45VTHHDHH63SU7MOU5ID5G667SKGCUWA, seg_pos={0 0}, upload=499401]: Uploading piece... Dec 29 09:06:17.919279061 (569h37m17.790362106s) [noise=true , address=45.91.116.12:4225 , piece_id=XA6Y42QOTAOYMUR446PX3O5ORN3N2WBQEBYP2E2AWFFZJKS33QBQ, seg_pos={0 0}, upload=499401]: Uploading piece... Dec 29 09:06:17.920072632 (569h37m17.791155667s) [noise=true , address=145.79.227.16:4912 , piece_id=WTNVBIE5ABUMTNE6VIQLQHU7NT3HLS6JMCH5O4QOTROOKWS7M4IQ, seg_pos={0 0}, upload=499401]: Uploading piece... Dec 29 09:06:19.620925929 (569h37m19.492008964s) [upload=499401]: Next returned nil segment Dec 29 09:06:19.620944174 (569h37m19.492027219s) [upload=499401]: Waiting for error group managing segments... Dec 29 09:06:19.894453007 (569h37m19.765536042s) [noise=true , address=145.79.227.39:4015 , piece_id=U6S7MRFZIMYU23UDY6AACJLGNA4ZFDEYZOMIHDR5TSIM5VQ4X74A, seg_pos={0 0}, upload=499401]: Done uploading piece. err: <nil> Dec 29 09:06:19.902152678 (569h37m19.773235713s) [noise=true , address=145.79.227.4:4185 , piece_id=TJZUBWWCSOU7L5C6XWOX45VTHHDHH63SU7MOU5ID5G667SKGCUWA, seg_pos={0 0}, upload=499401]: Done uploading piece. err: <nil> Dec 29 09:06:19.907889683 (569h37m19.778972718s) [noise=true , address=145.79.227.35:4483 , piece_id=SH255JUSIEBQNCYEQWYK27LSNTLWKZXU2SA64ZS4KTVBSNLK7UAQ, seg_pos={0 0}, upload=499401]: Done uploading piece. err: <nil> Dec 29 09:06:19.909116218 (569h37m19.780199253s) [noise=true , address=145.79.227.52:4445 , piece_id=N7GOJLI7A4Q5ZS6IWZHZKIL2MO4ZOT5ZYNGAAD2AO53KDIKTHOPA, seg_pos={0 0}, upload=499401]: Done uploading piece. err: <nil> Dec 29 09:06:19.909747790 (569h37m19.780830835s) [noise=true , address=145.79.227.16:4912 , piece_id=WTNVBIE5ABUMTNE6VIQLQHU7NT3HLS6JMCH5O4QOTROOKWS7M4IQ, seg_pos={0 0}, upload=499401]: Done uploading piece. err: <nil> Dec 29 09:06:19.910361830 (569h37m19.781444865s) [noise=true , address=145.79.227.45:4230 , piece_id=NGUPSZ57D7VFGTDGJG7TBPAGZR3RTZ6NRW2KVL2ABSHJFBQPW4RA, seg_pos={0 0}, upload=499401]: Done uploading piece. err: <nil> Dec 29 09:06:19.919054061 (569h37m19.790137096s) [noise=true , address=145.79.227.13:4802 , piece_id=26VYYSIDI62FHCDUR4PEUN4FRV7FWBVI3S2JFQIE4AEPC5AWYOHA, seg_pos={0 0}, upload=499401]: Done uploading piece. err: <nil> Dec 29 09:06:19.931812725 (569h37m19.80289576s) [noise=true , address=145.79.227.7:4295 , piece_id=RPACOVOHGZVJ6HTKB6WMQQQSP47U2SYCRU7Y4V6TL7XKKKCQZN7Q, seg_pos={0 0}, upload=499401]: Done uploading piece. err: <nil> Dec 29 09:06:19.934836894 (569h37m19.805919939s) [noise=true , address=145.79.227.34:4906 , piece_id=AAEC7LJK7ZDM3T66EKUUFCV7HLSVCMJXBGM7VC7JBEKZ3GCWXKPA, seg_pos={0 0}, upload=499401]: Done uploading piece. err: <nil> Dec 29 09:06:19.941883451 (569h37m19.812966496s) [noise=true , address=145.79.227.47:4265 , piece_id=X5CJZ5ANYB3FDQSGUNRD4IDZARZTH5UEWE73OYE7YH5HZ7YOZP7Q, seg_pos={0 0}, upload=499401]: Done uploading piece. err: <nil> Dec 29 09:06:20.113607792 (569h37m19.984690837s) [noise=true , address=77.90.31.12:4590 , piece_id=RSTOIMLZRTPMYTM4YJWFTP66EFBL3PAMSDSQ3UTQLCTHFE62MSMA, seg_pos={0 0}, upload=499401]: Done uploading piece. err: <nil> Dec 29 09:06:20.156726365 (569h37m20.02780941s) [noise=true , address=77.90.31.19:4841 , piece_id=POLYHLZNIWON6UYDXQMFGKNUCIYBUMYMZCQUOOQCH45CRAPFCEKQ, seg_pos={0 0}, upload=499401]: Done uploading piece. err: <nil> Dec 29 09:06:20.177569414 (569h37m20.048652449s) [noise=true , address=77.90.31.21:4877 , piece_id=IY45BWYJXV5RLWQGKZVMFO3NSVKHKC46ZNXUU4F4GTZE5TNDGLPA, seg_pos={0 0}, upload=499401]: Done uploading piece. err: <nil> Dec 29 09:06:20.177616093 (569h37m20.048699138s) [seg_pos={0 0}, upload=499401]: Segment reached optimal threshold of 13 pieces. Dec 29 09:06:20.177947433 (569h37m20.049030488s) [noise=true , address=45.91.116.12:4225 , piece_id=XA6Y42QOTAOYMUR446PX3O5ORN3N2WBQEBYP2E2AWFFZJKS33QBQ, seg_pos={0 0}, upload=499401]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 29 09:06:20.177959467 (569h37m20.049042512s) [noise=true , address=45.91.116.12:4225 , piece_id=XA6Y42QOTAOYMUR446PX3O5ORN3N2WBQEBYP2E2AWFFZJKS33QBQ, seg_pos={0 0}, upload=499401]: Long tail cancellation due to optimal threshold Dec 29 09:06:20.178031804 (569h37m20.04911485s) [noise=true , address=45.91.116.8:4046 , piece_id=7ZI3WICEE3CYLYNKCSG2JR6PIXPIXDULUTFKJW2HISSN4KE754XA, seg_pos={0 0}, upload=499401]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 29 09:06:20.178047794 (569h37m20.04913083s) [noise=true , address=45.91.116.8:4046 , piece_id=7ZI3WICEE3CYLYNKCSG2JR6PIXPIXDULUTFKJW2HISSN4KE754XA, seg_pos={0 0}, upload=499401]: Long tail cancellation due to optimal threshold Dec 29 09:06:20.178312118 (569h37m20.049395173s) [noise=true , address=45.91.116.13:4154 , piece_id=SBP7IEJE7QCMN5KPVXVMNS2JF5TM23DSKM7DPQGLSRIEYAGNJRUA, seg_pos={0 0}, upload=499401]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 09:06:20.178322077 (569h37m20.049405122s) [noise=true , address=45.91.116.13:4154 , piece_id=SBP7IEJE7QCMN5KPVXVMNS2JF5TM23DSKM7DPQGLSRIEYAGNJRUA, seg_pos={0 0}, upload=499401]: Long tail cancellation due to optimal threshold Dec 29 09:06:20.178349319 (569h37m20.049432364s) [seg_pos={0 0}, upload=499401]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:06:20.178454439 (569h37m20.049537494s) [upload=499401]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:06:20.178461703 (569h37m20.049544748s) [upload=499401]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:06:20.464418989 (569h37m20.335502034s) [upload=499401]: Done uploading segments... Dec 29 09:06:20.464441432 (569h37m20.335524477s) [upload=499401]: Upload finished. err: <nil> Duration: 30.003224455s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..X.......X............X.......................XX....X......] duration: 3.74468266s trace: https://jaeger.dev.storj.io/trace/2818f7531409d35c FAIL (2025-12-29 09:06:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.744715193s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 21.39788288s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 09:06:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 21.397911315s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...X...X.....XXX......X....X..XXX..X.....X..XX.X...X...XX.X.] duration: 5.693037559s trace: https://jaeger.dev.storj.io/trace/7c8e10939289a0cc FAIL (2025-12-29 09:06:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: BCI5SLXRTJHWEKL46PFCGMM6QXR4LEPC5F26UUNEYHB2UUN3R6BQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 6IUHCI5DCJH5C3EC3XZXCF3NGNTPC7UFEFL7IDSQNNL5HGSOUKCA) context deadline exceeded ======LOGS====== 6m1.469875499s) [upload=499384]: Uploading segments... Dec 29 09:05:01.598914938 (569h36m1.469997973s) [upload=499384]: Got next segment. Inline: false Dec 29 09:05:01.599151668 (569h36m1.470234713s) [upload=499384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:05:01.599162639 (569h36m1.470245684s) [upload=499384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:05:01.844667787 (569h36m1.715750842s) [seg_pos={0 0}, upload=499384]: Begin upload segment... Dec 29 09:05:01.844733552 (569h36m1.715816597s) [seg_pos={0 0}, upload=499384]: Done begin upload segment. Dec 29 09:05:01.844775091 (569h36m1.715858136s) [noise=true , address=45.91.116.12:4226 , piece_id=5UETRVXH6ATEBP7EJ2H74NPSKM5DFJ2IXLEGLLKBUWEIVNRFY4EQ, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.845024577 (569h36m1.716107632s) [noise=true , address=45.91.116.13:4156 , piece_id=EKJMIKA74NMSXRB7YAX6RLHPZVKKDUJNXTOKO2PL5ECO3H5V4OMA, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.845762722 (569h36m1.716845767s) [noise=true , address=145.79.227.52:4447 , piece_id=7LQOQLZ3IOQ3SUU7MIQWPUSEVFAHCHKHZ3P3QQPBGJWPHNM44Y2A, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.846394555 (569h36m1.71747768s) [noise=true , address=145.79.227.32:4834 , piece_id=6JPUKZFJFLWUKNPLMQOH2NKRSN6MG72TXI7L6FZNVUGWNLOCG42Q, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.847167186 (569h36m1.718250231s) [noise=true , address=145.79.227.13:4803 , piece_id=C2XHLCTSCF4CLXY72FCMI7OMYUP77GUO3PQGHD54R7ZILFH6DXUA, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.847736400 (569h36m1.718819445s) [noise=true , address=145.79.227.7:4294 , piece_id=DRE3SC2WCODIQLQEONN5D6HKNRY3HLQWTJPA2OWTU5IBWLLABXBQ, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.848370869 (569h36m1.719453924s) [noise=true , address=77.90.31.13:4627 , piece_id=6IUHCI5DCJH5C3EC3XZXCF3NGNTPC7UFEFL7IDSQNNL5HGSOUKCA, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.849297723 (569h36m1.720380788s) [noise=true , address=145.79.227.19:4079 , piece_id=HN2TNOLJZEF3QUZCWIUL2H5S45ID45MWEQSUAKI7ROUVSNYSGKCA, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.850021301 (569h36m1.721104346s) [noise=true , address=145.79.227.51:4411 , piece_id=TB2HOKZWUXBHK3FIRMZG3RTTC2J266WD5OM2PFBBA3AGANRWS57A, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.851199465 (569h36m1.72228251s) [noise=true , address=45.91.116.10:4084 , piece_id=O4VRA7ZOBPK54J64KTA4OPZYMABOZ2QJO7TBVCALDKJOBTZWVSGQ, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.851369589 (569h36m1.722452644s) [noise=true , address=77.90.31.12:4590 , piece_id=Y54LEEHLVONQ6373YYPCXRVIGZ74FHD4Y4ITGMDXJU7DK7EPHGMQ, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.852079270 (569h36m1.723162315s) [noise=true , address=77.90.31.10:4519 , piece_id=BCI5SLXRTJHWEKL46PFCGMM6QXR4LEPC5F26UUNEYHB2UUN3R6BQ, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.852854557 (569h36m1.723937602s) [noise=true , address=45.91.116.8:4046 , piece_id=4RANVLDX5BJU4LFXGANLVJKHU5ESLRRD3GDLDFBINP7HRUWOYR6Q, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.853489195 (569h36m1.72457224s) [noise=true , address=77.90.31.14:4662 , piece_id=U65JXZVOYVAYYF7UYTF4HW626L7NUQF3PSSVSEIQKDSJS7N4OIBA, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.854344854 (569h36m1.725427899s) [noise=true , address=145.79.227.37:4565 , piece_id=D3POYOJ22UKPTIRCC3FTFN7IOXDUSABYYWCVSR7OZWAK7FREU26Q, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.855117696 (569h36m1.726200741s) [noise=true , address=145.79.227.11:4731 , piece_id=FSHFK6KKRB2IWG637H4YVE5KXIX5VKTE7SLL4NSGWDJSUYDNCGCA, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:03.539696731 (569h36m3.410779776s) [upload=499384]: Next returned nil segment Dec 29 09:05:03.539717229 (569h36m3.410800274s) [upload=499384]: Waiting for error group managing segments... Dec 29 09:05:03.866365246 (569h36m3.737448291s) [noise=true , address=145.79.227.19:4079 , piece_id=HN2TNOLJZEF3QUZCWIUL2H5S45ID45MWEQSUAKI7ROUVSNYSGKCA, seg_pos={0 0}, upload=499384]: Done uploading piece. err: <nil> Dec 29 09:05:03.869686420 (569h36m3.740769495s) [noise=true , address=145.79.227.37:4565 , piece_id=D3POYOJ22UKPTIRCC3FTFN7IOXDUSABYYWCVSR7OZWAK7FREU26Q, seg_pos={0 0}, upload=499384]: Done uploading piece. err: <nil> Dec 29 09:05:03.870761537 (569h36m3.741844592s) [noise=true , address=145.79.227.51:4411 , piece_id=TB2HOKZWUXBHK3FIRMZG3RTTC2J266WD5OM2PFBBA3AGANRWS57A, seg_pos={0 0}, upload=499384]: Done uploading piece. err: <nil> Dec 29 09:05:03.871853546 (569h36m3.742936601s) [noise=true , address=145.79.227.52:4447 , piece_id=7LQOQLZ3IOQ3SUU7MIQWPUSEVFAHCHKHZ3P3QQPBGJWPHNM44Y2A, seg_pos={0 0}, upload=499384]: Done uploading piece. err: <nil> Dec 29 09:05:03.871909403 (569h36m3.742992458s) [noise=true , address=145.79.227.13:4803 , piece_id=C2XHLCTSCF4CLXY72FCMI7OMYUP77GUO3PQGHD54R7ZILFH6DXUA, seg_pos={0 0}, upload=499384]: Done uploading piece. err: <nil> Dec 29 09:05:03.893072782 (569h36m3.764155827s) [noise=true , address=145.79.227.7:4294 , piece_id=DRE3SC2WCODIQLQEONN5D6HKNRY3HLQWTJPA2OWTU5IBWLLABXBQ, seg_pos={0 0}, upload=499384]: Done uploading piece. err: <nil> Dec 29 09:05:03.918798647 (569h36m3.789881692s) [noise=true , address=145.79.227.32:4834 , piece_id=6JPUKZFJFLWUKNPLMQOH2NKRSN6MG72TXI7L6FZNVUGWNLOCG42Q, seg_pos={0 0}, upload=499384]: Done uploading piece. err: <nil> Dec 29 09:05:03.922181508 (569h36m3.793264563s) [noise=true , address=145.79.227.11:4731 , piece_id=FSHFK6KKRB2IWG637H4YVE5KXIX5VKTE7SLL4NSGWDJSUYDNCGCA, seg_pos={0 0}, upload=499384]: Done uploading piece. err: <nil> Dec 29 09:05:04.122500086 (569h36m3.993583131s) [noise=true , address=77.90.31.14:4662 , piece_id=U65JXZVOYVAYYF7UYTF4HW626L7NUQF3PSSVSEIQKDSJS7N4OIBA, seg_pos={0 0}, upload=499384]: Done uploading piece. err: <nil> Dec 29 09:05:04.137230614 (569h36m4.008313659s) [noise=true , address=45.91.116.8:4046 , piece_id=4RANVLDX5BJU4LFXGANLVJKHU5ESLRRD3GDLDFBINP7HRUWOYR6Q, seg_pos={0 0}, upload=499384]: Done uploading piece. err: <nil> Dec 29 09:05:04.147990394 (569h36m4.019073439s) [noise=true , address=77.90.31.13:4627 , piece_id=6IUHCI5DCJH5C3EC3XZXCF3NGNTPC7UFEFL7IDSQNNL5HGSOUKCA, seg_pos={0 0}, upload=499384]: Done uploading piece. err: <nil> Dec 29 09:05:04.182406206 (569h36m4.053489251s) [noise=true , address=77.90.31.10:4519 , piece_id=BCI5SLXRTJHWEKL46PFCGMM6QXR4LEPC5F26UUNEYHB2UUN3R6BQ, seg_pos={0 0}, upload=499384]: Done uploading piece. err: <nil> Dec 29 09:05:04.291661073 (569h36m4.162744118s) [noise=true , address=77.90.31.12:4590 , piece_id=Y54LEEHLVONQ6373YYPCXRVIGZ74FHD4Y4ITGMDXJU7DK7EPHGMQ, seg_pos={0 0}, upload=499384]: Done uploading piece. err: <nil> Dec 29 09:05:04.291688135 (569h36m4.16277118s) [seg_pos={0 0}, upload=499384]: Segment reached optimal threshold of 13 pieces. Dec 29 09:05:04.292069812 (569h36m4.163152857s) [noise=true , address=45.91.116.10:4084 , piece_id=O4VRA7ZOBPK54J64KTA4OPZYMABOZ2QJO7TBVCALDKJOBTZWVSGQ, seg_pos={0 0}, upload=499384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:40066->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:40066->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40066->45.91.116.10:4084: use of closed network connection Dec 29 09:05:04.292084591 (569h36m4.163167636s) [noise=true , address=45.91.116.10:4084 , piece_id=O4VRA7ZOBPK54J64KTA4OPZYMABOZ2QJO7TBVCALDKJOBTZWVSGQ, seg_pos={0 0}, upload=499384]: Long tail cancellation due to optimal threshold Dec 29 09:05:04.292108856 (569h36m4.163191991s) [noise=true , address=45.91.116.12:4226 , piece_id=5UETRVXH6ATEBP7EJ2H74NPSKM5DFJ2IXLEGLLKBUWEIVNRFY4EQ, seg_pos={0 0}, upload=499384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:42504->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:42504->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42504->45.91.116.12:4226: use of closed network connection Dec 29 09:05:04.292119597 (569h36m4.163202642s) [noise=true , address=45.91.116.12:4226 , piece_id=5UETRVXH6ATEBP7EJ2H74NPSKM5DFJ2IXLEGLLKBUWEIVNRFY4EQ, seg_pos={0 0}, upload=499384]: Long tail cancellation due to optimal threshold Dec 29 09:05:04.292353121 (569h36m4.163436166s) [noise=true , address=45.91.116.13:4156 , piece_id=EKJMIKA74NMSXRB7YAX6RLHPZVKKDUJNXTOKO2PL5ECO3H5V4OMA, seg_pos={0 0}, upload=499384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 29 09:05:04.292363231 (569h36m4.163446276s) [noise=true , address=45.91.116.13:4156 , piece_id=EKJMIKA74NMSXRB7YAX6RLHPZVKKDUJNXTOKO2PL5ECO3H5V4OMA, seg_pos={0 0}, upload=499384]: Long tail cancellation due to optimal threshold Dec 29 09:05:04.292445197 (569h36m4.163528262s) [seg_pos={0 0}, upload=499384]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:05:04.292582949 (569h36m4.163665994s) [upload=499384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:05:04.292589772 (569h36m4.163672817s) [upload=499384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:05:04.581739287 (569h36m4.452822332s) [upload=499384]: Done uploading segments... Dec 29 09:05:04.581766519 (569h36m4.452849564s) [upload=499384]: Upload finished. err: <nil> Duration: 5.693069369s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [X..X.......X............X.......................XX....X.....] duration: 3.956273373s trace: https://jaeger.dev.storj.io/trace/2818f7531409d35c FAIL (2025-12-29 09:05:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.956299241s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 37.872848015s trace: https://jaeger.dev.storj.io/trace/5d58dafcda5e27db FAIL (2025-12-29 09:05:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 07 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030106762 (569h10m36.901189797s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030404980 (569h10m36.901488015s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.030768121 (569h10m36.901851156s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031143476 (569h10m36.902226511s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031556453 (569h10m36.902639498s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.031893524 (569h10m36.902976559s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032331889 (569h10m36.903414924s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.032785082 (569h10m36.903868117s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033255808 (569h10m36.904338843s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.033904283 (569h10m36.904987328s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.035856500 (569h10m36.906939545s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.037541348 (569h10m36.908624383s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.044912965 (569h10m36.91599601s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:37.046492001 (569h10m36.917575036s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Uploading piece... Dec 29 08:39:38.331871497 (569h10m38.202954532s) [upload=499017]: Next returned nil segment Dec 29 08:39:38.331883530 (569h10m38.202966565s) [upload=499017]: Waiting for error group managing segments... Dec 29 08:39:38.661120276 (569h10m38.532203321s) [noise=true , address=64.49.13.170:4719 , piece_id=X6BA47DX22X5YICPYSQ4E2P6XHEFXSQYV5FKXRZ4F3LUN5ENGSZA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.691309391 (569h10m38.562392437s) [noise=true , address=64.49.13.138:4491 , piece_id=X2GCTHJ43CCMG6PYB243XJ644FY2J6TFUD5BUNYNFKLUOEBQVPNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.920469847 (569h10m38.791552893s) [noise=true , address=64.49.13.170:4727 , piece_id=VDFU6FRNZRL2GCQQAMVTTHAVM6R3BLOAEHXJYIYJLD7ALUFFOXKA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:38.946676097 (569h10m38.817759142s) [noise=true , address=145.79.227.43:4163 , piece_id=HK6YZXFJX3B2CJ7W5CIBZ6Z6MJTZZKESUBPHSN6RVUQUFLCDDQVQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.125416155 (569h10m38.99649919s) [noise=true , address=45.91.116.5:4320 , piece_id=SPQBNAJ5QLGOHFVKZQTSR4ECWVD6VIPYEGS7OSCAEQ2JQOFJWHGA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.139878214 (569h10m39.010961249s) [noise=true , address=45.91.116.11:4123 , piece_id=5KEVQQCQHRQNBOG4WJPNCWLO53RGAWGU6SYCPVMLYTAOC4XIIWXA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.165207634 (569h10m39.036290669s) [noise=true , address=77.90.31.10:4532 , piece_id=OIF6FHJTIZYMYAR5GFLD7KWGFWDQBI7KHNS6YWZHF3LL5GDFKF5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.200008551 (569h10m39.071091586s) [noise=true , address=77.90.31.11:4581 , piece_id=S5LJHVPNV7YYGUDPGJRCYGN7FBSBSGE3IFZRKVWFKYGHZ2SJ3GZQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.333912162 (569h10m39.204995197s) [noise=true , address=77.90.31.14:4679 , piece_id=EFPMSFK7BVDR7OGLJAHI2RLTJV4IA4KLP32HDI6XHS2675DYFQ4A, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.671961909 (569h10m39.543044944s) [noise=true , address=193.149.164.143:4694 , piece_id=EYEKKDWUIRS5FCJQORLZ7ZQB6RDIRAZ7U72E5RMZ2ALJAMD6VOEQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.761043584 (569h10m39.632126619s) [noise=true , address=77.90.31.15:4707 , piece_id=TRCOYQUFW5XKV663PE52N3ZTJ7WX2JWRP3CXS7FLJ67JQ3T7OA5Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.772551157 (569h10m39.643634202s) [noise=true , address=145.79.227.40:4078 , piece_id=U4MJSIUD7C4XVDSMI2SABUVDMGGMTB2AEDEVLHVVMOFXLLPVKSNQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.794693378 (569h10m39.665776413s) [noise=true , address=145.79.227.44:4218 , piece_id=AZGZWTLKU2FLAS43JOBSPUKK5ZTA53T6R4YLMG4PW22CYGCFBXMA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809452813 (569h10m39.680535858s) [noise=true , address=145.79.227.33:4899 , piece_id=SRBFAGTR2SCQFAFBT74EIAKRQEGLDKA67F7UHQ2HYFTEF67NZAYQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: <nil> Dec 29 08:39:39.809478311 (569h10m39.680561356s) [seg_pos={0 0}, upload=499017]: Segment reached optimal threshold of 14 pieces. Dec 29 08:39:39.809676128 (569h10m39.680759173s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.809683261 (569h10m39.680766316s) [noise=true , address=145.79.227.30:4775 , piece_id=SV6BZFCH7KOWL7RWFNJEIE2FO7XKORGRNRTLYLIQZV5ED4ZVYX2Q, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809821204 (569h10m39.680904249s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38874->64.49.13.169:4707: use of closed network connection Dec 29 08:39:39.809841032 (569h10m39.680924067s) [noise=true , address=64.49.13.169:4707 , piece_id=E3MCBDTHFMPS6BL7LPCNUMTUJCJDOQQDKL2BK5JCUQ3PY76ZKXIA, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.809862072 (569h10m39.680945158s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL): protocol: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42402->77.90.31.19:4872: use of closed network connection Dec 29 08:39:39.809872853 (569h10m39.680955898s) [noise=true , address=77.90.31.19:4872 , piece_id=2IZFTUWY56DYZE4WG7S27ZMAXNHUSMQUGUGNGQ6P6QU56B4MQ5VQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810107901 (569h10m39.681190946s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 29 08:39:39.810117248 (569h10m39.681200283s) [noise=true , address=145.79.227.49:4364 , piece_id=ST4ZMN4T7JTWEBEQ6FEYHIWKBZYQ2IHTZBCDSWWD55D3CFEVIUPQ, seg_pos={0 0}, upload=499017]: Long tail cancellation due to optimal threshold Dec 29 08:39:39.810158106 (569h10m39.681241141s) [seg_pos={0 0}, upload=499017]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 08:39:39.810253427 (569h10m39.681336463s) [upload=499017]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 08:39:39.810261202 (569h10m39.681344247s) [upload=499017]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 08:39:40.099441014 (569h10m39.970524069s) [upload=499017]: Done uploading segments... Dec 29 08:39:40.099467956 (569h10m39.970551001s) [upload=499017]: Upload finished. err: <nil> Duration: 37.872861722s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X...X...X.....XXX......X....X..XXX..X.....X..XX.X...X...XX.X] duration: 30.002243508s trace: https://jaeger.dev.storj.io/trace/7c8e10939289a0cc FAIL (2025-12-29 09:05:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: BCI5SLXRTJHWEKL46PFCGMM6QXR4LEPC5F26UUNEYHB2UUN3R6BQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 6IUHCI5DCJH5C3EC3XZXCF3NGNTPC7UFEFL7IDSQNNL5HGSOUKCA) context deadline exceeded ======LOGS====== 6m1.469875499s) [upload=499384]: Uploading segments... Dec 29 09:05:01.598914938 (569h36m1.469997973s) [upload=499384]: Got next segment. Inline: false Dec 29 09:05:01.599151668 (569h36m1.470234713s) [upload=499384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:05:01.599162639 (569h36m1.470245684s) [upload=499384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:05:01.844667787 (569h36m1.715750842s) [seg_pos={0 0}, upload=499384]: Begin upload segment... Dec 29 09:05:01.844733552 (569h36m1.715816597s) [seg_pos={0 0}, upload=499384]: Done begin upload segment. Dec 29 09:05:01.844775091 (569h36m1.715858136s) [noise=true , address=45.91.116.12:4226 , piece_id=5UETRVXH6ATEBP7EJ2H74NPSKM5DFJ2IXLEGLLKBUWEIVNRFY4EQ, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.845024577 (569h36m1.716107632s) [noise=true , address=45.91.116.13:4156 , piece_id=EKJMIKA74NMSXRB7YAX6RLHPZVKKDUJNXTOKO2PL5ECO3H5V4OMA, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.845762722 (569h36m1.716845767s) [noise=true , address=145.79.227.52:4447 , piece_id=7LQOQLZ3IOQ3SUU7MIQWPUSEVFAHCHKHZ3P3QQPBGJWPHNM44Y2A, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.846394555 (569h36m1.71747768s) [noise=true , address=145.79.227.32:4834 , piece_id=6JPUKZFJFLWUKNPLMQOH2NKRSN6MG72TXI7L6FZNVUGWNLOCG42Q, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.847167186 (569h36m1.718250231s) [noise=true , address=145.79.227.13:4803 , piece_id=C2XHLCTSCF4CLXY72FCMI7OMYUP77GUO3PQGHD54R7ZILFH6DXUA, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.847736400 (569h36m1.718819445s) [noise=true , address=145.79.227.7:4294 , piece_id=DRE3SC2WCODIQLQEONN5D6HKNRY3HLQWTJPA2OWTU5IBWLLABXBQ, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.848370869 (569h36m1.719453924s) [noise=true , address=77.90.31.13:4627 , piece_id=6IUHCI5DCJH5C3EC3XZXCF3NGNTPC7UFEFL7IDSQNNL5HGSOUKCA, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.849297723 (569h36m1.720380788s) [noise=true , address=145.79.227.19:4079 , piece_id=HN2TNOLJZEF3QUZCWIUL2H5S45ID45MWEQSUAKI7ROUVSNYSGKCA, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.850021301 (569h36m1.721104346s) [noise=true , address=145.79.227.51:4411 , piece_id=TB2HOKZWUXBHK3FIRMZG3RTTC2J266WD5OM2PFBBA3AGANRWS57A, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.851199465 (569h36m1.72228251s) [noise=true , address=45.91.116.10:4084 , piece_id=O4VRA7ZOBPK54J64KTA4OPZYMABOZ2QJO7TBVCALDKJOBTZWVSGQ, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.851369589 (569h36m1.722452644s) [noise=true , address=77.90.31.12:4590 , piece_id=Y54LEEHLVONQ6373YYPCXRVIGZ74FHD4Y4ITGMDXJU7DK7EPHGMQ, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.852079270 (569h36m1.723162315s) [noise=true , address=77.90.31.10:4519 , piece_id=BCI5SLXRTJHWEKL46PFCGMM6QXR4LEPC5F26UUNEYHB2UUN3R6BQ, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.852854557 (569h36m1.723937602s) [noise=true , address=45.91.116.8:4046 , piece_id=4RANVLDX5BJU4LFXGANLVJKHU5ESLRRD3GDLDFBINP7HRUWOYR6Q, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.853489195 (569h36m1.72457224s) [noise=true , address=77.90.31.14:4662 , piece_id=U65JXZVOYVAYYF7UYTF4HW626L7NUQF3PSSVSEIQKDSJS7N4OIBA, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.854344854 (569h36m1.725427899s) [noise=true , address=145.79.227.37:4565 , piece_id=D3POYOJ22UKPTIRCC3FTFN7IOXDUSABYYWCVSR7OZWAK7FREU26Q, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:01.855117696 (569h36m1.726200741s) [noise=true , address=145.79.227.11:4731 , piece_id=FSHFK6KKRB2IWG637H4YVE5KXIX5VKTE7SLL4NSGWDJSUYDNCGCA, seg_pos={0 0}, upload=499384]: Uploading piece... Dec 29 09:05:03.539696731 (569h36m3.410779776s) [upload=499384]: Next returned nil segment Dec 29 09:05:03.539717229 (569h36m3.410800274s) [upload=499384]: Waiting for error group managing segments... Dec 29 09:05:03.866365246 (569h36m3.737448291s) [noise=true , address=145.79.227.19:4079 , piece_id=HN2TNOLJZEF3QUZCWIUL2H5S45ID45MWEQSUAKI7ROUVSNYSGKCA, seg_pos={0 0}, upload=499384]: Done uploading piece. err: <nil> Dec 29 09:05:03.869686420 (569h36m3.740769495s) [noise=true , address=145.79.227.37:4565 , piece_id=D3POYOJ22UKPTIRCC3FTFN7IOXDUSABYYWCVSR7OZWAK7FREU26Q, seg_pos={0 0}, upload=499384]: Done uploading piece. err: <nil> Dec 29 09:05:03.870761537 (569h36m3.741844592s) [noise=true , address=145.79.227.51:4411 , piece_id=TB2HOKZWUXBHK3FIRMZG3RTTC2J266WD5OM2PFBBA3AGANRWS57A, seg_pos={0 0}, upload=499384]: Done uploading piece. err: <nil> Dec 29 09:05:03.871853546 (569h36m3.742936601s) [noise=true , address=145.79.227.52:4447 , piece_id=7LQOQLZ3IOQ3SUU7MIQWPUSEVFAHCHKHZ3P3QQPBGJWPHNM44Y2A, seg_pos={0 0}, upload=499384]: Done uploading piece. err: <nil> Dec 29 09:05:03.871909403 (569h36m3.742992458s) [noise=true , address=145.79.227.13:4803 , piece_id=C2XHLCTSCF4CLXY72FCMI7OMYUP77GUO3PQGHD54R7ZILFH6DXUA, seg_pos={0 0}, upload=499384]: Done uploading piece. err: <nil> Dec 29 09:05:03.893072782 (569h36m3.764155827s) [noise=true , address=145.79.227.7:4294 , piece_id=DRE3SC2WCODIQLQEONN5D6HKNRY3HLQWTJPA2OWTU5IBWLLABXBQ, seg_pos={0 0}, upload=499384]: Done uploading piece. err: <nil> Dec 29 09:05:03.918798647 (569h36m3.789881692s) [noise=true , address=145.79.227.32:4834 , piece_id=6JPUKZFJFLWUKNPLMQOH2NKRSN6MG72TXI7L6FZNVUGWNLOCG42Q, seg_pos={0 0}, upload=499384]: Done uploading piece. err: <nil> Dec 29 09:05:03.922181508 (569h36m3.793264563s) [noise=true , address=145.79.227.11:4731 , piece_id=FSHFK6KKRB2IWG637H4YVE5KXIX5VKTE7SLL4NSGWDJSUYDNCGCA, seg_pos={0 0}, upload=499384]: Done uploading piece. err: <nil> Dec 29 09:05:04.122500086 (569h36m3.993583131s) [noise=true , address=77.90.31.14:4662 , piece_id=U65JXZVOYVAYYF7UYTF4HW626L7NUQF3PSSVSEIQKDSJS7N4OIBA, seg_pos={0 0}, upload=499384]: Done uploading piece. err: <nil> Dec 29 09:05:04.137230614 (569h36m4.008313659s) [noise=true , address=45.91.116.8:4046 , piece_id=4RANVLDX5BJU4LFXGANLVJKHU5ESLRRD3GDLDFBINP7HRUWOYR6Q, seg_pos={0 0}, upload=499384]: Done uploading piece. err: <nil> Dec 29 09:05:04.147990394 (569h36m4.019073439s) [noise=true , address=77.90.31.13:4627 , piece_id=6IUHCI5DCJH5C3EC3XZXCF3NGNTPC7UFEFL7IDSQNNL5HGSOUKCA, seg_pos={0 0}, upload=499384]: Done uploading piece. err: <nil> Dec 29 09:05:04.182406206 (569h36m4.053489251s) [noise=true , address=77.90.31.10:4519 , piece_id=BCI5SLXRTJHWEKL46PFCGMM6QXR4LEPC5F26UUNEYHB2UUN3R6BQ, seg_pos={0 0}, upload=499384]: Done uploading piece. err: <nil> Dec 29 09:05:04.291661073 (569h36m4.162744118s) [noise=true , address=77.90.31.12:4590 , piece_id=Y54LEEHLVONQ6373YYPCXRVIGZ74FHD4Y4ITGMDXJU7DK7EPHGMQ, seg_pos={0 0}, upload=499384]: Done uploading piece. err: <nil> Dec 29 09:05:04.291688135 (569h36m4.16277118s) [seg_pos={0 0}, upload=499384]: Segment reached optimal threshold of 13 pieces. Dec 29 09:05:04.292069812 (569h36m4.163152857s) [noise=true , address=45.91.116.10:4084 , piece_id=O4VRA7ZOBPK54J64KTA4OPZYMABOZ2QJO7TBVCALDKJOBTZWVSGQ, seg_pos={0 0}, upload=499384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:40066->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:40066->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40066->45.91.116.10:4084: use of closed network connection Dec 29 09:05:04.292084591 (569h36m4.163167636s) [noise=true , address=45.91.116.10:4084 , piece_id=O4VRA7ZOBPK54J64KTA4OPZYMABOZ2QJO7TBVCALDKJOBTZWVSGQ, seg_pos={0 0}, upload=499384]: Long tail cancellation due to optimal threshold Dec 29 09:05:04.292108856 (569h36m4.163191991s) [noise=true , address=45.91.116.12:4226 , piece_id=5UETRVXH6ATEBP7EJ2H74NPSKM5DFJ2IXLEGLLKBUWEIVNRFY4EQ, seg_pos={0 0}, upload=499384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:42504->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:42504->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42504->45.91.116.12:4226: use of closed network connection Dec 29 09:05:04.292119597 (569h36m4.163202642s) [noise=true , address=45.91.116.12:4226 , piece_id=5UETRVXH6ATEBP7EJ2H74NPSKM5DFJ2IXLEGLLKBUWEIVNRFY4EQ, seg_pos={0 0}, upload=499384]: Long tail cancellation due to optimal threshold Dec 29 09:05:04.292353121 (569h36m4.163436166s) [noise=true , address=45.91.116.13:4156 , piece_id=EKJMIKA74NMSXRB7YAX6RLHPZVKKDUJNXTOKO2PL5ECO3H5V4OMA, seg_pos={0 0}, upload=499384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 29 09:05:04.292363231 (569h36m4.163446276s) [noise=true , address=45.91.116.13:4156 , piece_id=EKJMIKA74NMSXRB7YAX6RLHPZVKKDUJNXTOKO2PL5ECO3H5V4OMA, seg_pos={0 0}, upload=499384]: Long tail cancellation due to optimal threshold Dec 29 09:05:04.292445197 (569h36m4.163528262s) [seg_pos={0 0}, upload=499384]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:05:04.292582949 (569h36m4.163665994s) [upload=499384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:05:04.292589772 (569h36m4.163672817s) [upload=499384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:05:04.581739287 (569h36m4.452822332s) [upload=499384]: Done uploading segments... Dec 29 09:05:04.581766519 (569h36m4.452849564s) [upload=499384]: Upload finished. err: <nil> Duration: 30.002265981s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.X..X.......X............X.......................XX....X....] duration: 3.484486225s trace: https://jaeger.dev.storj.io/trace/2818f7531409d35c FAIL (2025-12-29 09:05:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.484548864s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)